Summary of Reward Learning From Preference with Ties, by Jinsong Liu et al.
Reward Learning From Preference With Ties
by Jinsong Liu, Dongdong Ge, Ruihao Zhu
First submitted to arxiv on: 5 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper addresses a crucial aspect of Reinforcement Learning from Human Feedback (RLHF) by introducing the Bradley-Terry model with ties (BTT). The BTT model captures human preferences more accurately than the traditional Bradley-Terry (BT) model, which neglects tied preferences. By incorporating ties, the paper shows that preference strength measurement can be biased if ties are disregarded. Experiments validate the benefits of using BTT, and fine-tuning with BTT outperforms BT on synthetic datasets labeled by state-of-the-art language models. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper helps improve Reinforcement Learning from Human Feedback by introducing a new model that takes into account when people don’t have a clear preference between two options. This is important because ignoring these “ties” can make the model’s decisions biased. The researchers tested their new approach and found it works better than the old way of doing things. |
Keywords
» Artificial intelligence » Fine tuning » Reinforcement learning from human feedback » Rlhf
