Summary of Mitigating Forgetting in Llm Supervised Fine-tuning and Preference Learning, by Heshan Fernando et al.
Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning
by Heshan Fernando, Han Shen, Parikshit Ram, Yi Zhou, Horst Samulowitz, Nathalie Baracaldo, Tianyi Chen
First submitted to arxiv on: 20 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Optimization and Control (math.OC); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a novel approach to post-training large language models (LLMs) for effective and safe applications. The traditional method involves sequential supervised fine-tuning (SFT) and preference learning (RLHF or DPO), but this has been shown to be sub-optimal due to the LLM forgetting its initial training. To address this, the authors theoretically prove the sub-optimality of sequential post-training and propose a joint framework that empirically outperforms the traditional approach while having similar computational costs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models (LLMs) are crucial for many applications, but their training requires careful consideration to ensure they remain effective and safe. The traditional method involves two stages: supervised fine-tuning (SFT) and preference learning (RLHF or DPO). However, this approach has been shown to have limitations. In this paper, the authors present a new way of training LLMs that avoids these issues. |
Keywords
» Artificial intelligence » Fine tuning » Rlhf » Supervised
