Summary of [cls] Attention Is All You Need For Training-free Visual Token Pruning: Make Vlm Inference Faster, by Qizhe Zhang et al.
[CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster
by Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, Shanghang Zhang
First submitted to arxiv on: 2 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This study addresses the inefficiency of large vision-language models (VLMs) that rely on numerous visual tokens when interacting with large language models (LLMs). The authors find that pruning visual tokens based on inaccurate cross-attentions between text and visual tokens in LLMs leads to significant performance degradation. To overcome this issue, they propose FasterVLM, a training-free method that evaluates the importance of visual tokens more accurately by utilizing attentions between the [CLS] token and image tokens from the visual encoder. This approach eliminates redundant visual tokens immediately after the visual encoder, resulting in faster VLM inference while maintaining 90% of the performance of LLaVA-1.5-7B with a 95% reduction ratio. The authors demonstrate FasterVLM’s effectiveness on various VLM architectures and reduction ratios, outperforming existing text-visual attention-based methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making computer vision models work faster by getting rid of unnecessary information. Currently, these models rely on a lot of visual data when talking to language models, which slows them down. The researchers found that using the wrong method to figure out what information is important leads to bad results. They developed a new way to prioritize information called FasterVLM, which works without needing to train the model from scratch. This approach eliminates unnecessary information right after it’s created, making the overall process faster and more efficient. The authors tested this method on different models and showed that it outperforms existing methods. |
Keywords
» Artificial intelligence » Attention » Encoder » Inference » Pruning » Token
