Summary of Neptune: the Long Orbit to Benchmarking Long Video Understanding, by Arsha Nagrani et al.
Neptune: The Long Orbit to Benchmarking Long Video Understanding
by Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hornung, Nitesh Bharadwaj Gundavarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, Mikhail Sirotenko, Yukun Zhu, Tobias Weyand
First submitted to arxiv on: 12 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Neptune, a benchmark for long video understanding, is introduced to challenge current models and spur the development of more advanced ones. It requires reasoning over long time horizons and across different modalities, unlike existing datasets that focus on short clips. To address the limitations of manual annotation at high cost, a scalable dataset creation pipeline leveraging large language models (LLMs) and vision-language models (VLMs) is proposed. This pipeline generates dense, time-aligned video captions and tough question-answer decoy sets for segments up to 15 minutes in length. Neptune covers long video reasoning abilities, including multimodal reasoning, and provides a new open-source model-based metric GEM to score open-ended responses. Evaluations show that most current models perform poorly on Neptune, particularly on questions testing temporal ordering, counting, and state changes. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Neptune is a special kind of test for computers that can understand long videos. Right now, many video understanding tests are focused on short parts of videos (just 10-30 seconds). These tests often rely on powerful image recognition models that look at each frame separately, rather than considering the whole video. To make testing longer videos more efficient and accurate, Neptune uses large language and vision models to automatically create captions for long videos and tricky question-answer pairs. This test is important because it helps developers create better computers that can understand what’s happening in a video over a longer period of time. |
