Summary of Interleaved-modal Chain-of-thought, by Jun Gao et al.
Interleaved-Modal Chain-of-Thought
by Jun Gao, Yongqi Li, Ziqiang Cao, Wenjie Li
First submitted to arxiv on: 29 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Interleaved-modal Chain-of-Thought (ICoT) method generates sequential reasoning steps with paired visual and textual rationales for vision-language models (VLMs). ICoT requires VLMs to enable fine-grained interleaved-modal content, which current VLMs struggle to fulfill. To realize ICoT over existing VLMs, the Attention-driven Selection (ADS) strategy is proposed, which inserts regions of the input image into the reasoning steps with negligible additional latency. ADS relies solely on the attention map of VLMs without requiring parameterization and can be generalized to various VLM architectures. The method is evaluated on three benchmarks, achieving substantial performance improvements (up to 14%) and interpretability enhancements compared to existing multimodal CoT prompting methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary ICoT is a new way for language models to explain their thinking. It’s like taking notes while solving a puzzle, but instead of writing words, the model writes both text and images that relate to each other. This helps us understand how the model arrived at its answer. The problem is that current image-based models aren’t good at explaining themselves in this way. To fix this, the researchers developed a technique called ADS (Attention-driven Selection) that takes the attention map of the model and uses it to decide what parts of an image are most important for the explanation. This allows existing models to generate explanations like ICoT without needing changes. |
Keywords
» Artificial intelligence » Attention » Prompting
