Summary of Sbora: Low-rank Adaptation with Regional Weight Updates, by Lai-man Po et al.
SBoRA: Low-Rank Adaptation with Regional Weight Updates
by Lai-Man Po, Yuyang Liu, Haoxuan Wu, Tianqi Zhang, Wing-Yin Yu, Zhuohan Wang, Zeyu Jiang, Kun Li
First submitted to arxiv on: 7 Jul 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Computation and Language (cs.CL); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces Standard Basis LoRA (SBoRA), a novel parameter-efficient fine-tuning approach for Large Language Models that builds upon the pioneering works of Low-Rank Adaptation (LoRA) and Orthogonal Adaptation. SBoRA reduces the number of trainable parameters by half or doubles the rank with similar learning performance as LoRA, while improving learning performance. By utilizing orthogonal standard basis vectors to initialize one of the low-rank matrices, SBoRA facilitates regional weight updates and memory-efficient fine-tuning. This results in two variants, SBoRA-FA and SBoRA-FB, where only one of the matrices is updated, leading to a sparse update matrix with predominantly zero rows or columns. The paper demonstrates the superiority of SBoRA-FA over LoRA in various fine-tuning tasks, including commonsense reasoning and arithmetic reasoning. Code is available at this URL. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary SBoRA is a new way to make language models more efficient. It works by using special math tricks to reduce the number of things that need to be learned. This makes it faster and uses less memory. The paper shows that SBoRA is better than another method called LoRA at doing some tasks, like understanding simple math problems. |
Keywords
* Artificial intelligence * Fine tuning * Lora * Low rank adaptation * Parameter efficient
