Summary of Moving Off-the-grid: Scene-grounded Video Representations, by Sjoerd Van Steenkiste et al.
Moving Off-the-Grid: Scene-Grounded Video Representations
by Sjoerd van Steenkiste, Daniel Zoran, Yi Yang, Yulia Rubanova, Rishabh Kabra, Carl Doersch, Dilara Gokay, Joseph Heyward, Etienne Pot, Klaus Greff, Drew A. Hudson, Thomas Albert Keck, Joao Carreira, Alexey Dosovitskiy, Mehdi S. M. Sajjadi, Thomas Kipf
First submitted to arxiv on: 8 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A self-supervised video representation model, Moving Off-the-Grid (MooG), is proposed, which enables tokens to move off the grid and represent scene elements consistently as they move across the image plane through time. The model disentangles representation structure and image structure using cross-attention and positional embeddings. A simple self-supervised objective, next frame prediction, trained on video data, results in a set of latent tokens that bind to specific scene structures and track them as they move. MooG’s learned representation is demonstrated to be useful for various downstream tasks, outperforming “on-the-grid” baselines. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary MooG is a new way to make computers understand videos better. Instead of using a fixed grid, the model lets tokens move around to represent different parts of the scene as they change over time. This helps computers learn to recognize things like people or objects moving in a video. The model does this by looking at what’s happening frame by frame and trying to predict what will happen next. It’s shown to be good at helping with various tasks, like recognizing what’s happening in a video. |
Keywords
» Artificial intelligence » Cross attention » Self supervised
