Summary of Efficient Vision-language Models by Summarizing Visual Tokens Into Compact Registers, By Yuxin Wen et al.
Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers
by Yuxin Wen, Qingqing Cao, Qichen Fu, Sachin Mehta, Mahyar Najibi
First submitted to arxiv on: 17 Oct 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Visual Compact Token Registers (Victor) method reduces the number of visual tokens in vision-language models, enabling faster training and inference times without sacrificing performance. By summarizing visual information into a smaller set of register tokens, Victor decreases computational overhead while maintaining model accuracy. The approach adds learnable register tokens after visual tokens and discards them after a few layers, allowing for efficient processing of complex reasoning tasks on images. This paper demonstrates the effectiveness of Victor by showing a 43% reduction in training time and a 3.3X boost in inference throughput with only an 8-token register set (about 1% of the original tokens), resulting in less than a 4% accuracy drop. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you have a superpower that lets you understand images better. Recently, machines got closer to having this power thanks to special models called vision-language models. These models are great at doing complex tasks on pictures. But, they need lots of computer power to do it fast and efficiently. The problem is that these models use many “visual tokens” to understand what’s in the picture. This makes them slow and uses too much energy. To fix this, scientists created a new way to store visual information called Visual Compact Token Registers (Victor). Victor takes lots of visual tokens and turns them into just a few special tokens that are easy for the model to use. This makes the model work faster and more efficiently without losing its ability to understand images well. |
Keywords
» Artificial intelligence » Inference » Token
