Summary of Investigating Regularization Of Self-play Language Models, by Reda Alami et al.
Investigating Regularization of Self-Play Language Models
by Reda Alami, Abdalgader Abubaker, Mastane Achab, Mohamed El Amine Seddik, Salem Lahlou
First submitted to arxiv on: 4 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper investigates how different forms of regularization affect language model alignment using self-play. The study focuses on three approaches: reinforcement learning from human feedback (RLHF), direct preference optimization (DPO), and self-play fine-tuning (SPIN). While SPIN is efficient, it presents performance instability issues during the learning phase, which can be mitigated by playing against a mixture of previous iterates. To address this issue, the authors propose incorporating Kullback-Leibler regularization to maintain proximity to the reference policy, and using fictitious play to smooth the opponent policy across iterations. The authors demonstrate the effectiveness of these approaches on MT-Bench and the Hugging Face Open LLM Leaderboard. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how different methods affect language models learning from themselves. They compare three ways: getting feedback from humans (RLHF), making decisions based on human preferences (DPO), and letting the model play with itself (SPIN). SPIN is helpful, but it has a problem where it can get stuck in a bad place during training. To fix this, they suggest two ideas: adding a special kind of regularization to keep the model close to its original self, or using an old trick called fictitious play to make the model be more reasonable. They tested these ideas and saw that they work well on some language tasks. |
Keywords
» Artificial intelligence » Alignment » Fine tuning » Language model » Optimization » Regularization » Reinforcement learning from human feedback » Rlhf
