Summary of Learning Dynamics Of Llm Finetuning, by Yi Ren et al.
Learning Dynamics of LLM Finetuning
by Yi Ren, Danica J. Sutherland
First submitted to arxiv on: 15 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed framework for learning dynamics in large language models offers a powerful tool for understanding the behavior of deep learning systems during different types of fine-tuning. By analyzing the step-wise decomposition of influence accumulation among potential responses, researchers can gain insights into why specific types of hallucination are strengthened after fine-tuning. The framework also provides a novel perspective on LLM’s fine-tuning and inspires a simple method to improve alignment performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are becoming increasingly powerful tools for understanding human behavior and generating responses. This paper studies how these models learn during different types of training, including instruction tuning and preference tuning. Researchers found that certain types of hallucination, or incorrect information, are strengthened after fine-tuning. They also discovered a unique “squeezing effect” that can make desired outputs less likely if the model is trained for too long. Overall, this framework provides new insights into how large language models work and how they can be improved. |
Keywords
» Artificial intelligence » Alignment » Deep learning » Fine tuning » Hallucination » Instruction tuning
