Summary of Computational Limits Of Low-rank Adaptation (lora) For Transformer-based Models, by Jerry Yao-chieh Hu et al.
Computational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models
by Jerry Yao-Chieh Hu, Maojiang Su, En-Jui Kuo, Zhao Song, Han Liu
First submitted to arxiv on: 5 Jun 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computational Complexity (cs.CC); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed study explores the computational limitations of Low-Rank Adaptation (LoRA) updates for fine-tuning transformer-based models. The researchers observe that low-rank decompositions within the gradient computation of LoRA adaptation can lead to algorithmic speedups, allowing them to identify a phase transition behavior and prove the existence of nearly linear algorithms. They achieve this by controlling the LoRA update computation term by term, assuming the Strong Exponential Time Hypothesis (SETH). The study also derives a shared upper bound threshold for specific norms resulting from input sequences, pretrained weights, and adapter matrices, showing that efficient approximation algorithms exist only below this threshold. Additionally, they prove the existence of nearly linear approximation algorithms for LoRA adaptation by utilizing hierarchical low-rank structures of LoRA gradients and approximating gradients with chained low-rank approximations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary LoRA is a way to fine-tune transformer-based models. Researchers studied how LoRA works and found that it can be faster if they use special math tricks. They discovered a “phase transition” where LoRA gets really fast or really slow, depending on the numbers involved. They also proved that there are shortcuts (nearly linear algorithms) to make LoRA even faster. To show this works, they looked at two real-life scenarios: adapting some parts of attention heads and adapting all parts. |
Keywords
» Artificial intelligence » Attention » Fine tuning » Lora » Low rank adaptation » Transformer
