Summary of Lora Training in the Ntk Regime Has No Spurious Local Minima, by Uijeong Jang et al.
LoRA Training in the NTK Regime has No Spurious Local Minima
by Uijeong Jang, Jason D. Lee, Ernest K. Ryu
First submitted to arxiv on: 19 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Optimization and Control (math.OC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper delves into the theoretical understanding of Low-rank adaptation (LoRA) for fine-tuning large language models (LLMs). Researchers have long used LoRA to efficiently adapt these massive models, but the underlying theory was unclear. This study bridges that gap by analyzing LoRA in the neural tangent kernel (NTK) regime with a dataset containing N points. The findings indicate that: first, full fine-tuning without LoRA yields a low-rank solution of rank r ≤ √N; second, using LoRA with rank r ≥ √N eliminates spurious local minima, allowing gradient descent to find the low-rank solutions; and third, these found solutions generalize well. This work has significant implications for the development and optimization of language models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks at how we can make big language models smaller without losing their powers. Right now, we use something called Low-rank adaptation (LoRA) to do this, but we don’t fully understand why it works. The researchers in this paper did some math problems to figure out what’s going on. They found that when we fine-tune these massive models without LoRA, they can still be made smaller and better. Then, they used LoRA to make them even smaller and more powerful, eliminating any weird minima they might find along the way. And the best part? These smaller models are just as good at understanding language as the bigger ones. |
Keywords
* Artificial intelligence * Fine tuning * Gradient descent * Lora * Low rank adaptation * Optimization
