Summary of Avg-llava: a Large Multimodal Model with Adaptive Visual Granularity, by Zhibin Lan et al.
AVG-LLaVA: A Large Multimodal Model with Adaptive Visual Granularity
by Zhibin Lan, Liqiang Niu, Fandong Meng, Wenbo Li, Jie Zhou, Jinsong Su
First submitted to arxiv on: 20 Sep 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces AVG-LLaVA, a new type of Large Language Model (LMM) that can adaptively select the appropriate visual granularity for processing high-resolution images. Unlike traditional LMMs, which divide images into multiple local and global tokens, AVG-LLaVA reduces the number of tokens by selecting the most relevant level of detail. The model consists of two modules: a visual granularity scaler and a router. The scaler generates tokens at different granularities using pooling layers, while the router selects the best granularity based on the image and instruction. A novel training paradigm called RGLF is also proposed to align the router’s predictions with the LMM’s preferences. Experimental results show that AVG-LLaVA outperforms other models across 11 benchmarks, reducing visual tokens by up to 85.3% and speeding up inference by up to 2.53 times. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research paper presents a new way for computers to process images with lots of detail. Most computer models break down images into many small pieces, which can be slow and not very accurate. The new model, called AVG-LLaVA, does the opposite – it looks at the image as a whole and only breaks it down when necessary. This makes the processing faster and more accurate. The team also came up with a new way to train this model without needing lots of extra data. In tests, the new model performed better than others on many different tasks. |
Keywords
» Artificial intelligence » Inference » Large language model
