Summary of Accelerating Multimodal Large Language Models Via Dynamic Visual-token Exit and the Empirical Findings, by Qiong Wu et al.
Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
by Qiong Wu, Wenhao Lin, Weihao Ye, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji
First submitted to arxiv on: 29 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG); Multimedia (cs.MM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed method, dynamic visual-token exit (DyVTE), aims to improve the efficiency of Multimoal Large Language Models (MLLMs) by addressing the observed visual redundancy in existing models. By analyzing the attention behaviors of MLLMs, researchers identify three main inference stages: early fusion, intra-modality modeling, and multimodal reasoning. They reveal that visual tokens stop contributing to reasoning when text tokens receive enough image information, leading to obvious visual redundancy. DyVTE uses lightweight hyper-networks to perceive text token status and decide the removal of all visual tokens after a certain layer, effectively addressing this issue. The method is validated through experiments on various benchmarks, including LLaVA, VILA, Eagle, and InternVL. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper proposes a new method called dynamic visual-token exit (DyVTE) to make large language models more efficient. Right now, these models use too many “visual tokens” which makes them slow and wasteful. The researchers looked at how these models work and found that they can stop using those extra visual tokens once they’ve gotten the information they need from text tokens. To do this, DyVTE uses a special network to see what’s happening with the text tokens and decide when it’s safe to stop using the visual tokens. This makes the model run faster and use less computer power. |
Keywords
» Artificial intelligence » Attention » Inference » Token
