Summary of Exploring Perceptual Limitation Of Multimodal Large Language Models, by Jiarui Zhang et al.
Exploring Perceptual Limitation of Multimodal Large Language Models
by Jiarui Zhang, Jinyi Hu, Mahyar Khayatkhoei, Filip Ilievski, Maosong Sun
First submitted to arxiv on: 12 Feb 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper investigates the limitations of Multimodal Large Language Models (MLLMs) in answering visual questions about small objects in images. The authors quantitatively study the perception of small objects in several state-of-the-art MLLMs and identify four independent factors that contribute to this limitation: object quality, size, distractors, and location. Surprisingly, they find that lower object quality, smaller object size, and certain visual distractions can significantly reduce MLLMs’ question-answering accuracy. This study contributes new evaluation protocols for analyzing the perception of future MLLMs and releases code and data to facilitate further investigations. The paper’s findings have implications for developing more accurate and robust MLLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research explores how well large language models can answer questions about small objects in pictures. Right now, these models are great at answering visual questions, but not as good when the objects are tiny. Scientists wanted to figure out why this is happening. They found four things that make it harder for the models to answer questions: if the object is low-quality, too small, or surrounded by distractions. They also found that where the object is in the picture matters. This study helps us understand what’s going on and how we can make these models better at answering questions about small objects. |
Keywords
* Artificial intelligence * Question answering
