Summary of Sail Into the Headwind: Alignment Via Robust Rewards and Dynamic Labels Against Reward Hacking, by Paria Rashidinejad et al.
Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking
by Paria Rashidinejad, Yuandong Tian
First submitted to arxiv on: 12 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Statistics Theory (math.ST); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper investigates a common issue in AI system development called “reward hacking,” where optimizing an imperfect reward model leads to undesirable behaviors. The authors identify two types of reward hacking: Type I, which is caused by subpar choices appearing more favorable, and Type II, which is caused by decent choices appearing less favorable. They show that many preference optimization methods suffer from both types of reward hacking. To mitigate these issues, the authors propose POWER, a new preference optimization method that combines weighted entropy with a robust reward maximization objective. POWER enjoys finite-sample guarantees under general function approximation and competes with the best-covered policy in the data. The authors also develop a novel technique to dynamically update preference labels toward certain “stationary labels,” resulting in diminishing gradients for untrustworthy samples. Empirically, POWER with dynamic labels (POWER-DL) outperforms state-of-the-art methods on alignment benchmarks and improves or maintains performance on downstream tasks like mathematical reasoning. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper talks about a problem called “reward hacking” that happens when we try to make AI systems more intelligent. This problem makes the AI do things we don’t want it to do. The authors looked at two kinds of reward hacking and found that many ways we use to make AI better are actually making it worse. They came up with a new way called POWER that can help fix this problem by making sure the AI doesn’t get stuck in bad behaviors. They also developed a technique to update what the AI thinks is good or bad, so it doesn’t get tricked into doing things we don’t want. |
Keywords
» Artificial intelligence » Alignment » Optimization
