Summary of Online Self-preferring Language Models, by Yuanzhao Zhai et al.
Online Self-Preferring Language Models
by Yuanzhao Zhai, Zhuo Zhang, Kele Xu, Hanyang Peng, Yue Yu, Dawei Feng, Cheng Yang, Bo Ding, Huaimin Wang
First submitted to arxiv on: 23 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel approach to large language models (LLMs) that leverages human preference datasets more effectively. The authors introduce Online Self-Preferring (OSP) language models that learn from self-generated response pairs and self-judged preference strengths. This allows OSP to explicitly model preference strength information, which is crucial for distinguishing different response pairs. To achieve this, the paper proposes a ranked pairing method and soft-preference cross-entropy loss. The results show that OSP achieves state-of-the-art alignment performance across various metrics in two widely used human preference datasets, while being more robust than RLHF when limited offline data are available. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us understand how we can make language models better at understanding what people like and dislike. Right now, these models learn from feedback that’s hard to get. To fix this, the authors came up with a new way called Online Self-Prefering (OSP). OSP lets the model create its own responses and then choose which ones it likes best. This helps the model understand how strong someone’s preference is for one response over another. The paper shows that this approach works better than others and can even learn to improve itself without needing more help from humans. |
Keywords
» Artificial intelligence » Alignment » Cross entropy » Rlhf
