Summary of Schema: State Changes Matter For Procedure Planning in Instructional Videos, by Yulei Niu et al.
SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos
by Yulei Niu, Wenliang Guo, Long Chen, Xudong Lin, Shih-Fu Chang
First submitted to arxiv on: 3 Mar 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates procedure planning in instructional videos by learning a goal-oriented sequence of action steps from partial visual state observations. Unlike previous works which focused solely on sequence modeling, this research emphasizes the importance of states in procedures, introducing State CHangEs MAtter (SCHEMA) for more structured state spaces. The authors leverage large language models to represent step states and track changes via cross-modal contrastive learning. They demonstrate their proposed SCHEMA model achieves state-of-the-art performance on CrossTask, COIN, and NIV benchmark datasets. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about teaching machines how to make a plan for doing something new. Imagine you’re watching a video that shows someone cooking a recipe, and the machine needs to learn what steps it should take to follow along. The researchers are trying to figure out how to do this by looking at pictures of the steps being taken and using special language models to understand what’s happening. They want to make sure their plan is good by testing it on different recipes and making sure it makes sense. |
