Summary of Preserving Pre-trained Representation Space: on Effectiveness Of Prefix-tuning For Large Multi-modal Models, by Donghoon Kim et al.
Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models
by Donghoon Kim, Gusang Lee, Kyuhong Shim, Byonghyo Shim
First submitted to arxiv on: 29 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper explores the effects of parameter-efficient fine-tuning (PEFT) on Large Multi-modal Models (LMMs), focusing on the strengths and weaknesses of various tuning strategies. It finds that model parameter tuning methods like LoRA and Adapters distort the feature representation space learned during pre-training, limiting the utilization of pre-trained knowledge. In contrast, prefix-tuning excels at preserving the representation space despite its lower performance on downstream tasks. The paper proposes a simple two-step PEFT strategy called Prefix-Tuned PEFT (PT-PEFT), which combines the benefits of both. Experimental results show that PT-PEFT improves performance in image captioning and visual question answering compared to vanilla PEFT methods, while preserving the representation space of pre-trained models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how Large Multi-modal Models are used for different tasks. It wants to know what happens when we make these models better for a specific job. The researchers found that some ways of making them better change how the model works, which isn’t good. They also discovered that another way makes the model work well, but not as good at the actual task. So they came up with a new way to make the model better that combines both. It worked really well and helped keep the original knowledge intact. |
Keywords
* Artificial intelligence * Fine tuning * Image captioning * Lora * Multi modal * Parameter efficient * Question answering
