Summary of Bridging the Gap Between Low-rank and Orthogonal Adaptation Via Householder Reflection Adaptation, by Shen Yuan et al.
Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation
by Shen Yuan, Haotian Liu, Hongteng Xu
First submitted to arxiv on: 24 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Householder reflection adaptation (HRA) method bridges the gap between low-rank and orthogonal adaptation techniques, efficiently adapting large-scale pre-training models in specific tasks or domains. By multiplying each frozen weight matrix with an orthogonal matrix constructed by a chain of learnable Householder reflections, HRA fine-tunes layers while maintaining model capacity and regularity. Regularizing orthogonality impacts the method’s performance, leading to different implementations. Compared to state-of-the-art methods, HRA achieves superior performance with fewer learnable parameters when adapting large language models and conditional image generators. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper proposes a new way to adapt pre-trained models for specific tasks or domains. It combines two techniques: low-rank adaptation and orthogonal adaptation. The method works by multiplying each weight matrix with an orthogonal matrix made up of small pieces of learnable parameters. This helps fine-tune the model’s layers while keeping it regular and efficient. The paper shows that this new method, called Householder reflection adaptation (HRA), performs better than other methods in adapting large language models and generating images. |
Keywords
» Artificial intelligence » Low rank adaptation
