Summary of Back to Basics: Revisiting Reinforce Style Optimization For Learning From Human Feedback in Llms, by Arash Ahmadian et al.
Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
by Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
First submitted to arxiv on: 22 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper explores the role of Reinforcement Learning from Human Feedback (RLHF) in achieving high performance in large language models. The recent literature has positioned Proximal Policy Optimization (PPO) as the canonical method for RL, but it comes with a high computational cost and requires sensitive hyperparameter tuning. In contrast, this study advocates for a less computationally expensive method that preserves and even increases performance. By revisiting the formulation of alignment from human preferences in the context of RL, the authors show that many components of PPO are unnecessary in an RLHF context and that simpler REINFORCE-style optimization variants outperform both PPO and newly proposed methods such as DPO and RAFT. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper looks at how we can make large language models better by using something called Reinforcement Learning from Human Feedback (RLHF). Right now, people are saying that a method called Proximal Policy Optimization (PPO) is the best way to do this. But PPO takes up a lot of computer power and requires tweaking lots of settings just right. The researchers in this study think that’s too much fuss for RLHF and suggest using something simpler instead. They looked at how we can use human preferences to make language models better and found that some things we thought were important aren’t actually necessary. They also showed that simpler ways of doing things are actually better than PPO. |
Keywords
* Artificial intelligence * Alignment * Hyperparameter * Optimization * Reinforcement learning from human feedback * Rlhf
