Summary of Streamingbench: Assessing the Gap For Mllms to Achieve Streaming Video Understanding, by Junming Lin et al.
StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
by Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, Maosong Sun
First submitted to arxiv on: 6 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel paper introduces StreamingBench, a benchmark designed to evaluate the real-time video understanding capabilities of Multimodal Large Language Models (MLLMs). The benchmark assesses three core aspects: visual understanding, omni-source understanding, and contextual understanding. It features 18 tasks with 900 videos and 4,500 human-curated QA pairs, simulating a continuous streaming scenario. Experiments on StreamingBench with 13 MLLMs reveal that even advanced models like Gemini 1.5 Pro and GPT-4o perform below human-level capabilities. The paper highlights the limitations of current MLLMs in real-time video understanding and aims to facilitate further advancements. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper introduces a new way to test how well computers can understand videos in real-time, similar to how humans watch TV or YouTube videos. It creates a special set of challenges called StreamingBench that includes 18 different tasks with 900 videos and many questions. The goal is to see how well computer models can answer these questions as they are watching the video, just like we do when we’re streaming a show or movie. |
Keywords
» Artificial intelligence » Gemini » Gpt
