Summary of Selective Self-rehearsal: a Fine-tuning Approach to Improve Generalization in Large Language Models, by Sonam Gupta et al.
Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models
by Sonam Gupta, Yatin Nandwani, Asaf Yehudai, Mayank Mishra, Gaurav Pandey, Dinesh Raghu, Sachindra Joshi
First submitted to arxiv on: 7 Sep 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Large language models (LLMs) are fine-tuned for specific tasks by training them on target datasets, which often leads to overfitting. Overfitting causes the model to become too specialized in either the task or the characteristics of the training data, resulting in a loss of generalization. To address this issue, researchers introduce Selective Self-Rehearsal (SSR), a fine-tuning approach that achieves performance comparable to standard supervised fine-tuning (SFT) while improving generalization. SSR leverages the fact that there can be multiple valid responses to a query and reduces model specialization during the fine-tuning stage. To achieve this, SSR first identifies the correct model responses from the training set by deploying an appropriate LLM as a judge. Then, it fine-tunes the model using the correct model responses and the gold response for the remaining samples. The effectiveness of SSR is demonstrated through experiments on identifying unanswerable queries across various datasets. The results show that standard SFT can lead to an average performance drop of up to 16.7% on multiple benchmarks such as MMLU and TruthfulQA, while SSR results in a close to 2% drop on average, indicating better generalization capabilities compared to standard SFT. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary LLMs are trained for specific tasks by fine-tuning them on target datasets, which often leads to overfitting. Overfitting makes the model too specialized and less generalizable. To fix this, a new approach called Selective Self-Rehearsal (SSR) is introduced. SSR helps the model learn from its own correct responses and reduces specialization during training. Experiments show that using SSR improves performance without sacrificing generalization. For example, on some benchmarks, standard fine-tuning can drop by 16.7%, while SSR only drops by 2%. This means SSR is better at handling different types of questions and staying accurate. |
Keywords
» Artificial intelligence » Fine tuning » Generalization » Overfitting » Supervised
