Summary of Bridging Compressed Image Latents and Multimodal Large Language Models, by Chia-hao Kao et al.
Bridging Compressed Image Latents and Multimodal Large Language Models
by Chia-Hao Kao, Cheng Chien, Yu-Jen Tseng, Yi-Hsin Chen, Alessandro Gnutti, Shao-Yuan Lo, Wen-Hsiao Peng, Riccardo Leonardi
First submitted to arxiv on: 29 Jul 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Machine Learning (cs.LG); Multimedia (cs.MM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents the first study on adapting compressed image latents for downstream vision tasks using Multimodal Large Language Models (MLLMs). MLLMs have successfully extended language models to images, but their large scale makes them difficult to deploy on resource-constrained devices. To address this, the authors propose a novel framework that adapts compressed image latents for MLLM-based vision tasks, excluding the entire downstream MLLM except part of its visual encoder from training. The proposed framework is general and applicable to various MLLMs, neural image codecs, and multiple application scenarios. Extensive experiments show that the method achieves great rate-accuracy performance with much less complexity. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper helps us understand how to make images smaller so they can be sent quickly over the internet. Right now, it takes a lot of energy and time to send big images from devices like cameras or smartphones. The authors came up with a new way to compress images using special neural networks that work well with big language models. They tested their method on different types of neural networks and showed that it works really well. |
Keywords
» Artificial intelligence » Encoder
