Summary of Mora: High-rank Updating For Parameter-efficient Fine-tuning, by Ting Jiang et al.
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
by Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
First submitted to arxiv on: 20 May 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper analyzes the limitations of low-rank adaptation in fine-tuning large language models (LLMs) using LoRA. It suggests that this mechanism hinders LLMs from effectively learning new knowledge. To address this, the authors propose MoRA, a high-rank updating method that maintains the same number of trainable parameters as LoRA. MoRA employs square matrices and non-parameter operators to reduce input dimensions and increase output dimensions. The authors evaluate their method on five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory, and pretraining. MoRA outperforms LoRA in memory-intensive tasks and achieves comparable performance on other tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low-rank adaptation is a way to make large language models more efficient. Researchers found that this method might not be the best way for these models to learn new things. To fix this, they created a new method called MoRA. MoRA uses special matrices to help the model learn and remember better. The authors tested their method on five tasks and found it worked better than LoRA in some cases. |
Keywords
» Artificial intelligence » Fine tuning » Instruction tuning » Lora » Low rank adaptation » Pretraining
