Summary of Self-play Fine-tuning Converts Weak Language Models to Strong Language Models, by Zixiang Chen and Yihe Deng and Huizhuo Yuan and Kaixuan Ji and Quanquan Gu
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
by Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu
First submitted to arxiv on: 2 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A novel approach to fine-tuning Large Language Models (LLMs) is proposed, which leverages the power of human-annotated data without requiring additional annotation. The method, called Self-Play fIne-tuNing (SPIN), starts from a supervised fine-tuned model and uses a self-play mechanism to refine its capabilities. This involves generating training data from previous iterations and discerning self-generated responses from those obtained from human-annotated data. SPIN progressively elevates the LLM, unlocking the full potential of human-annotated demonstration data for Supervised Fine-Tuning (SFT). Theoretical proofs demonstrate that the global optimum is achieved when the LLM policy aligns with the target data distribution. Empirical evaluations on benchmark datasets such as HuggingFace Open LLM Leaderboard, MT-Bench, and Big-Bench show significant performance improvements. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary A team of researchers has found a new way to make large language models better without needing lots of human help. They call it Self-Play fIne-tuNing (SPIN). It starts with a model that’s already been taught some things, and then makes the model play games against itself. This helps the model get even smarter by learning from its own mistakes. The team tested SPIN on lots of different datasets and found that it works really well, even better than other methods that use human help. |
Keywords
* Artificial intelligence * Fine tuning * Supervised
