Summary of Gradient Dynamics For Low-rank Fine-tuning Beyond Kernels, by Arif Kerem Dayi et al.
Gradient dynamics for low-rank fine-tuning beyond kernels
by Arif Kerem Dayi, Sitan Chen
First submitted to arxiv on: 23 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Statistics Theory (math.ST); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper investigates LoRA, a popular method for fine-tuning foundation models with minimal computational requirements and memory usage. LoRA modifies the pre-trained model’s weights by adding a low-rank perturbation, which is trained on supervised data for a specific task. Although LoRA has shown promising results, its underlying mathematical principles are still not well understood. The paper aims to shed light on what learning mechanisms drive gradient descent to converge to useful low-rank perturbations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study looks into LoRA, a way to make pre-trained models better for specific tasks using just a little extra computing power and memory. LoRA adds small changes to the original model’s weights, which are trained on labeled data for that task. Despite being successful, it’s unclear why this works mathematically. The researchers want to figure out what makes LoRA good at finding useful changes. |
Keywords
» Artificial intelligence » Fine tuning » Gradient descent » Lora » Supervised
