Summary of Vega: Learning Interleaved Image-text Comprehension in Vision-language Large Models, by Chenyu Zhou et al.
VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models
by Chenyu Zhou, Mengdan Zhang, Peixian Chen, Chaoyou Fu, Yunhang Shen, Xiawu Zheng, Xing Sun, Rongrong Ji
First submitted to arxiv on: 14 Jun 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Interleaved Image-Text Comprehension (IITC), a new task that challenges Multi-modal Large Models (MLLMs) to navigate complex scenarios involving irrelevant information. The IITC task requires models to accurately answer questions and follow instructions while disregarding superfluous elements in both images and text. To support this task, the authors create the VEGA dataset, tailored for scientific content, and a subtask called Image-Text Association (ITA) to refine image-text correlation skills. Evaluation of various MLLMs on the IITC task shows that even advanced models like Gemini-1.5-pro and GPT4V achieved only modest success. By employing a multi-task, multi-scale post-training strategy, the authors set a robust baseline for MLLMs on the IITC task, achieving an 85.8% accuracy rate in image association and a 0.508 Rouge score. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps machines understand complex pictures and words by creating new challenges for large language models. It’s like asking someone to find specific information in a big book with many irrelevant pages. The authors create special training data and test models on this task, showing that even the best models struggle. They also share how they improved their model’s performance using a multi-step training approach. |
Keywords
» Artificial intelligence » Gemini » Multi modal » Multi task » Rouge
