Summary of Asft: Aligned Supervised Fine-tuning Through Absolute Likelihood, by Ruoyu Wang et al.
ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood
by Ruoyu Wang, Jiachen Sun, Shaowei Hua, Quan Fang
First submitted to arxiv on: 14 Sep 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary In this paper, the authors propose Aligned Supervised Fine-Tuning (ASFT), a novel method for enhancing model performance by directly optimizing for the preferences or rankings of outcomes. The proposed approach aims to address limitations in Direct Preference Optimization (DPO) and its variants, particularly their sensitivity to Supervised Fine-Tuning (SFT). ASFT optimizes absolute likelihood for each response, eliminating the need for a reference model. Through theoretical gradient analysis, the authors demonstrate that ASFT mitigates issues with DPO’s loss function. The method is evaluated on instruction-following benchmarks and traditional text generation metrics using the Llama3 model fine-tuned on UltraFeedback and HH-RLHF datasets. Results show that ASFT consistently outperforms existing methods. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper proposes a new way to make language models better by directly optimizing for what people like or dislike. Currently, these models are trained to minimize errors, but this can lead to mistakes if the model is not aligned with human preferences. The authors introduce Aligned Supervised Fine-Tuning (ASFT), which improves upon existing methods and ensures that the model produces desired responses. They tested ASFT on various tasks and showed that it performs better than previous approaches. |
Keywords
* Artificial intelligence * Fine tuning * Likelihood * Loss function * Optimization * Rlhf * Supervised * Text generation
