Summary of Sparse Attention Vectors: Generative Multimodal Model Features Are Discriminative Vision-language Classifiers, by Chancharik Mitra et al.
Sparse Attention Vectors: Generative Multimodal Model Features Are Discriminative Vision-Language Classifiers
by Chancharik Mitra, Brandon Huang, Tianning Chai, Zhiqiu Lin, Assaf Arbelle, Rogerio Feris, Leonid Karlinsky, Trevor Darrell, Deva Ramanan, Roei Herzig
First submitted to arxiv on: 28 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed approach leverages generative large multimodal models (LMMs) for foundational discriminative vision-language tasks by extracting useful features from the model’s latent space. A finetuning-free method called Sparse Attention Vectors (SAVs) utilizes sparse attention head activations in LLMs as strong features for vision-language tasks, achieving state-of-the-art performance on a collection of discriminative tasks with only few-shot examples. This approach demonstrates robustness and scalability, generalizing to similar tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Generative Large Multimodal Models (LMMs) are really good at doing lots of things like describing pictures or answering questions about them. But they’re not so great when it comes to simple tasks like recognizing what’s in a picture. To fix this, scientists came up with a new way to use LMMs that works better for these kinds of tasks. They called it Sparse Attention Vectors (SAVs). It’s a special trick that uses only a small part of the model to get good results. And it worked! They tested it on lots of different tasks and it did really well, even when they didn’t have many examples to train it with. |
Keywords
» Artificial intelligence » Attention » Few shot » Latent space
