Summary of Mm-spubench: Towards Better Understanding Of Spurious Biases in Multimodal Llms, by Wenqian Ye et al.
MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs
by Wenqian Ye, Guangtao Zheng, Yunsheng Ma, Xu Cao, Bolin Lai, James M. Rehg, Aidong Zhang
First submitted to arxiv on: 24 Jun 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the problem of spurious bias in multimodal Large Language Models (MLLMs), which integrate vision and language models for joint understanding. The authors analyze the biases that occur when a vision model’s biases cascade into the alignment between visual and text tokens in MLLMs, leading to reliance on spurious correlations. A new benchmark, MM-SpuBench, is introduced for evaluating MLLMs’ reliance on these biases, using a visual question-answering (VQA) task with five open-source image datasets. The authors evaluate state-of-the-art MLLMs and find that they persistently rely on spurious correlations, highlighting the need for new methods to mitigate these biases. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper looks at how deep learning models can be tricked into thinking certain things are connected just because they’re both shown together. These models use images and words to understand each other, but sometimes this leads to mistakes. The authors create a special test to see if these models are making mistakes by relying on unrelated information. They find that the best models still make these mistakes, so we need new ways to stop them. |
Keywords
» Artificial intelligence » Alignment » Deep learning » Question answering
