Summary of Asymmetry in Low-rank Adapters Of Foundation Models, by Jiacheng Zhu et al.
Asymmetry in Low-Rank Adapters of Foundation Models
by Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, Justin Solomon
First submitted to arxiv on: 26 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper focuses on optimizing large pre-trained foundation models by fine-tuning a subset of parameters using Low-Rank Adaptation (LoRA), which is particularly effective. The authors investigate the role of LoRA matrices during fine-tuning and find that the B matrix has a distinct function in creating desired outputs, whereas the A matrix extracts features from inputs. They demonstrate that fine-tuning B is more effective than fine-tuning A, and that a random untrained A performs similarly to a fine-tuned one. The authors also bound the generalization of low-rank adapters using an information-theoretic lens, showing that parameter savings improve the bound. They support their conclusions with experiments on RoBERTa, BART-Large, LLaMA-2, and ViTs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how to make large pre-trained models better by updating just a few of its parameters. The authors found that certain parts of these models are more important than others for getting the right results. They show that some parts can even be replaced with random numbers without much loss. This is because those parts are not as important for the model’s overall performance. The authors also tested this idea on four different types of models and showed that it works. |
Keywords
* Artificial intelligence * Fine tuning * Generalization * Llama * Lora * Low rank adaptation
