Summary of Vpo: Leveraging the Number Of Votes in Preference Optimization, by Jae Hyeon Cho et al.
VPO: Leveraging the Number of Votes in Preference Optimization
by Jae Hyeon Cho, Minkyung Park, Byung-Jun Lee
First submitted to arxiv on: 30 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Medium Difficulty Summary: This paper introduces Direct Preference Optimization (DPO) and its extension to Vote-based Preference Optimization (VPO), which trains language models using human preference data. Unlike Reinforcement Learning from Human Feedback (RLHF), DPO bypasses the explicit reward modeling phase by iterating over sentence pairs in a preference dataset. The authors create these datasets through voting processes involving multiple individuals, which can provide insight into the subjective nature of human preferences. By employing the Bayesian Minimum Mean Square Error (Bayesian MMSE) estimator to model the probability that one generation is preferable to another, VPO incorporates the number of votes on both sides to distinguish between controversial and obvious generation pairs. The authors demonstrate that these proposed algorithms outperform various existing methods, including their base algorithms. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low Difficulty Summary: This paper makes it possible for computers to learn from what people like and dislike about text. It’s called Direct Preference Optimization (DPO) and allows the computer to create new text by using human preferences. Instead of trying to figure out what people want, DPO just looks at what they like and don’t like, and tries to create more text that matches those preferences. The authors also introduce a new way to use voting data from multiple people to make sure the computer creates text that is liked by most people. |
Keywords
» Artificial intelligence » Optimization » Probability » Reinforcement learning from human feedback » Rlhf
