Summary of Uncertainty-aware Reward Model: Teaching Reward Models to Know What Is Unknown, by Xingzhou Lou et al.
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
by Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, Junge Zhang
First submitted to arxiv on: 1 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Uncertainty-aware Reward Model (URM) and its ensemble variant, URME, aim to improve the alignment of large language models (LLMs) with human expectations by capturing both aleatoric and epistemic uncertainties. The URM uses a probabilistic value head to model the distribution of disentangled human preference attributes, while the URME examines discrepancies among individual URM models within the ensemble to quantify epistemic uncertainty. Experimental results demonstrate that URM outperforms competitive large-scale models on RewardBench and enhances LLMs’ generation quality through best-of-n sampling (BoN), iterative direct preference optimization (iterative DPO), and proximal policy optimization (PPO). The study highlights the importance of reliable reward predictions with lower uncertainty, which result in higher-quality alignment. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Large language models are designed to generate human-like text. However, these models need guidance on what is good or bad content. The Uncertainty-aware Reward Model helps give this guidance by considering how uncertain humans are about their preferences. This model works better than others because it captures the different ways people might think about something. The results show that this approach leads to better language generation and a more accurate understanding of human preferences. |
Keywords
» Artificial intelligence » Alignment » Optimization
