Summary of Av-odyssey Bench: Can Your Multimodal Llms Really Understand Audio-visual Information?, by Kaixiong Gong et al.
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?
by Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, Xiangyu Yue
First submitted to arxiv on: 3 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multimedia (cs.MM); Sound (cs.SD); Audio and Speech Processing (eess.AS)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed DeafTest reveals that multimodal large language models (MLLMs) struggle with simple tasks like determining which sound is louder or has a higher pitch. To address this limitation, the AV-Odyssey Bench is introduced as a comprehensive audio-visual benchmark to evaluate MLLMs’ understanding of audio-visual information. The benchmark consists of 4,555 carefully crafted problems incorporating text, visual, and audio components, requiring models to effectively leverage clues from both inputs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Multimodal large language models can understand many things, like conversations and images. But they’re not perfect. They struggle with simple tasks that humans find easy, like comparing the loudness or pitch of two sounds. To help MLLMs get better, a new test called AV-Odyssey Bench was created. It has thousands of problems that combine text, pictures, and sound. Models must use clues from both to answer questions correctly. |
