Summary of Direct Nash Optimization: Teaching Language Models to Self-improve with General Preferences, by Corby Rosset et al.
Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
by Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie
First submitted to arxiv on: 4 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel approach to improving large language models (LLMs) using preference feedback from an oracle. The traditional method, Reinforcement Learning from Human Feedback (RLHF), is limited by its reliance on point-wise rewards that fail to capture complex preferences. Instead, the authors introduce Direct Nash Optimization (DNO), a scalable and provable algorithm that optimizes general preferences through contrastive learning. DNO uses a regression-based objective and enjoys monotonic improvement across iterations, allowing it to surpass even strong teachers like GPT-4. In experiments, a 7B parameter Orca-2.5 model aligned by DNO achieved the state-of-the-art win-rate against GPT-4-Turbo on AlpacaEval 2.0. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps big language models get better at understanding what we like and don’t like. Right now, it’s hard to tell them what we prefer because rewards are too simple. The authors came up with a new way to make the model learn from our feedback, called Direct Nash Optimization (DNO). It’s fast, reliable, and can even outdo super smart teachers! They tested it on some really big models and showed that it works much better than other approaches. |
Keywords
* Artificial intelligence * Gpt * Optimization * Regression * Reinforcement learning from human feedback * Rlhf
