Summary of Aim: Adaptive Inference Of Multi-modal Llms Via Token Merging and Pruning, by Yiwu Zhong et al.
AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning
by Yiwu Zhong, Zhuoming Liu, Yin Li, Liwei Wang
First submitted to arxiv on: 4 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach to reducing the computational demands of large language models (LLMs) is proposed in this work, enabling their deployment in resource-constrained environments and for long-context tasks. The method, adaptive inference, iteratively merges tokens based on embedding similarity before feeding them into LLMs, and prunes tokens within LLM layers based on multi-modal importance. This minimalist design can be applied to both video and image LLMs, yielding a substantial reduction in computation load (e.g., 7-fold) while preserving performance. Experimental results demonstrate state-of-the-art performance in long video understanding (+4.6 on MLVU), outperforming existing methods under similar computational costs. The study provides insights into token redundancy and LLM layer behaviors, guiding future research in designing efficient multi-modal LLMs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models can understand images and videos really well, but they need a lot of computing power to do it. This makes them hard to use on devices with limited resources or for tasks that require processing long sequences of data. Researchers have found a way to make these models more efficient without sacrificing their performance. They did this by developing an adaptive inference method that can be used with both video and image language models. The new method works by merging similar tokens together before the model processes them, and then removing unimportant tokens from the model itself. This reduced the amount of computing power needed to process the data by a lot (7 times less in some cases). The method also performed better than other existing methods for processing long videos. The study provides insights into how this works and could help guide future research in making language models even more efficient. |
Keywords
» Artificial intelligence » Embedding » Inference » Multi modal » Token
