Summary of Compcap: Improving Multimodal Large Language Models with Composite Captions, by Xiaohui Chen et al.
CompCap: Improving Multimodal Large Language Models with Composite Captions
by Xiaohui Chen, Satya Narayan Shukla, Mahmoud Azab, Aashu Singh, Qifan Wang, David Yang, ShengYun Peng, Hanchao Yu, Shen Yan, Xuewen Zhang, Baosheng He
First submitted to arxiv on: 6 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Multimodal Large Language Models (MLLMs) are challenged when interpreting composite images (CIs), which are synthetic visuals created by merging multiple visual elements. While CIs are common in real-world applications, MLLM research has primarily focused on natural images. Our study reveals that current MLLMs struggle to accurately understand CIs, often failing to extract information or perform complex reasoning. We attribute this gap to the lack of high-quality image-caption datasets for CIs and introduce Composite Captions (CompCap), a flexible framework that leverages LLMs and automation tools to synthesize CIs with accurate captions. CompCap-118K, a dataset containing 118K image-caption pairs across six CI types, is introduced. Supervised fine-tuning of MLLMs shows significant improvements in understanding CIs, yielding average gains of 1.7%, 2.0%, and 2.9% across eleven benchmarks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about how well computers can understand images that are made by combining different pieces of visual information. These kinds of images are common in real life, but researchers have mostly focused on understanding natural images. We found that current computer models struggle to understand these combined images and make sense of them. To help with this problem, we created a new way to generate captions for these images, called Composite Captions (CompCap). We also made a big dataset with 118K image-caption pairs to train our models. By testing our approach, we found that it can greatly improve how well computers understand combined images. |
Keywords
» Artificial intelligence » Fine tuning » Supervised
