Summary of Multi-modal Auto-regressive Modeling Via Visual Words, by Tianshuo Peng et al.
Multi-modal Auto-regressive Modeling via Visual Words
by Tianshuo Peng, Zuchao Li, Lefei Zhang, Hai Zhao, Ping Wang, Bo Du
First submitted to arxiv on: 12 Mar 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper introduces Large Multi-modal Models (LMMs), which extend Large Language Models’ (LLMs) capabilities to process visual information. By proposing the concept of visual tokens, mapping visual features to probability distributions over LLM’s vocabulary, the authors provide supervision information for visual modelling. The approach is validated through results and ablation studies on 5 VQA tasks and 4 benchmark toolkits, demonstrating its powerful performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper shows how to make computers better at understanding images by combining two types of AI models: language and image processing. They create a new way to translate visual information into something the computer can understand, called “visual tokens”. This helps the computer learn from both text and images together, which is important for tasks like recognizing objects in pictures. The results show that this approach works well on several different tasks. |
Keywords
» Artificial intelligence » Multi modal » Probability
