Summary of Sok-bench: a Situated Video Reasoning Benchmark with Aligned Open-world Knowledge, by Andong Wang et al.
SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge
by Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Li Erran Li, Chuang Gan
First submitted to arxiv on: 15 May 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper tackles the challenge of learning commonsense reasoning from visual contexts and scenes in real-world scenarios, a crucial step towards advanced artificial intelligence. The authors highlight that existing video reasoning benchmarks are limited as they were designed for factual or situated reasoning, failing to incorporate broader knowledge in real-world settings. To address this gap, the researchers propose SOK-Bench, a new benchmark consisting of 44K questions and 10K situations with instance-level annotations depicted in videos. The benchmark requires reasoning processes to understand and apply situated knowledge and general knowledge for problem-solving. To generate such a dataset, the authors develop an automatic and scalable method using large language models (LLMs) and multi-layered language models (MLLMs). The paper also presents insightful conclusions from evaluating recent mainstream large vision-language models on the SOK-Bench benchmark. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about teaching computers to understand common sense and make decisions based on what they see. Right now, computers are good at recognizing things in pictures or videos, but they’re not very good at understanding what’s really going on. The researchers created a new test called SOK-Bench that challenges computers to use their common sense to solve problems. They did this by creating 44,000 questions and 10,000 scenarios with answers that require the computer to understand both what it sees and broader knowledge. |
