Summary of Mmie: Massive Multimodal Interleaved Comprehension Benchmark For Large Vision-language Models, by Peng Xia et al.
MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
by Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, Lijuan Wang, Huaxiu Yao
First submitted to arxiv on: 14 Oct 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces MMIE, a large-scale benchmark for evaluating Large Vision-Language Models (LVLMs) in multimodal comprehension and generation. The existing benchmarks are limited by their scale, scope, and evaluation depth, while current metrics are costly or biased. MMIE addresses these challenges with 20K meticulously curated multimodal queries across various categories, fields, and subfields. It supports both interleaved inputs and outputs, offering multiple-choice and open-ended question formats to evaluate diverse competencies. The paper also proposes a reliable automated evaluation metric, leveraging a scoring model fine-tuned with human-annotated data and systematic evaluation criteria. Extensive experiments demonstrate the effectiveness of MMIE in providing a comprehensive evaluation of LVLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about creating a new way to test machines that can understand and create text and images together. Right now, there’s no good way to measure how well these machines do this task, so the authors created a big dataset with 20,000 questions that mix text and images in different ways. The dataset has questions from many different categories like math, science, and art. They also came up with a new way to score the answers from the machines that’s fair and accurate. When they tested this new way of scoring, they found that even the best machines don’t do very well yet. |
