Summary of Rethinking Reward Model Evaluation: Are We Barking Up the Wrong Tree?, by Xueru Wen et al.
Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?
by Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, Le Sun
First submitted to arxiv on: 8 Oct 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research paper investigates Reward Models (RMs) in language models and explores the relationship between RM accuracy and downstream policy performance. The authors employ a synthetic setting to examine how differences in RM accuracy translate into gaps in optimized policy performance, revealing a weak positive correlation between accuracy and performance. However, they find that policies optimized towards RMs with similar accuracy can exhibit distinct performance, highlighting the inadequacy of relying solely on accuracy to reflect its impact on policy optimization. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Reward Models (RMs) are important for making language models align with human preferences. Currently, we measure RM accuracy by comparing it to a set of preference data that has been manually annotated. While this method is simple and widely used, we don’t fully understand how RM accuracy affects the performance of policies that have been optimized. In this study, we use a fake setting to see how differences in RM accuracy affect gaps in policy performance. We found that while there’s a weak link between accuracy and performance, policies that have been optimized towards RMs with similar accuracy can still perform differently. This shows that relying only on accuracy is not enough to understand its impact on policy optimization. |
Keywords
* Artificial intelligence * Optimization
