Summary of Training-free Mitigation Of Language Reasoning Degradation After Multimodal Instruction Tuning, by Neale Ratzlaff et al.
Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning
by Neale Ratzlaff, Man Luo, Xin Su, Vasudev Lal, Phillip Howard
First submitted to arxiv on: 4 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed research investigates the effects of multimodal instruction tuning on the language reasoning capabilities of powerful large language models (LLMs). The study focuses on LLaVA, a leading multimodal framework that integrates LLMs like Vicuna or Mistral with the CLIP vision encoder. The researchers compare the performance of original LLMs with their multimodal-adapted counterparts across eight language reasoning tasks. The results show that the impact of multimodal learning varies between Vicuna and Mistral, with Vicuna showing improvements in most tasks and Mistral experiencing a degradation in language reasoning capabilities. Additionally, the study finds that while multimodal instruction tuning consistently degrades performance on mathematical reasoning tasks, it enhances performance on commonsense reasoning tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper explores how combining large language models (LLMs) with vision encoders affects their ability to reason about language. The researchers test two types of LLMs, Vicuna and Mistral, and find that one improves while the other gets worse at language tasks when combined with a vision encoder. They also discover that this combination helps or hurts different types of language reasoning tasks in different ways. |
Keywords
» Artificial intelligence » Encoder » Instruction tuning
