Summary of Interpreting Language Reward Models Via Contrastive Explanations, by Junqi Jiang et al.
Interpreting Language Reward Models via Contrastive Explanations
by Junqi Jiang, Tom Bewley, Saumitra Mishra, Freddy Lecue, Manuela Veloso
First submitted to arxiv on: 25 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This research proposes a novel approach to explaining the behavior of reward models (RMs) in large language models (LLMs). RMs are crucial for aligning LLM outputs with human values by predicting and comparing reward scores. However, current RMs are “black boxes” whose predictions are not explainable. The proposed method uses contrastive explanations to characterize an RM’s local behavior by generating a diverse set of new comparisons that modify manually specified high-level evaluation attributes. This allows for the investigation of global sensitivity of RMs to each evaluation attribute and the extraction of representative examples to explain and compare behaviors. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This study aims to make large language models more trustworthy by explaining how they make decisions. Reward models are important because they help align the model’s outputs with what humans consider valuable. But right now, these models are like black boxes – we don’t know why they make certain predictions. The researchers propose a new way to explain how reward models work by creating many different scenarios that test the model’s behavior. This helps us understand how the model is influenced by certain factors and makes it easier to compare the behaviors of different models. |
