Summary of When Can Proxies Improve the Sample Complexity Of Preference Learning?, by Yuchen Zhu et al.
When Can Proxies Improve the Sample Complexity of Preference Learning?
by Yuchen Zhu, Daniel Augusto de Souza, Zhengyan Shi, Mengyue Yang, Pasquale Minervini, Alexander D’Amour, Matt J. Kusner
First submitted to arxiv on: 21 Dec 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Medium Difficulty summary: This paper tackles the issue of “reward hacking” in Large Language Models (LLMs), where maximizing a proxy reward does not necessarily increase the true reward. Existing methods use various tricks, such as regularization and reward hacking detectors, to limit the influence of proxy preferences on LLMs. However, when expert data is available, it’s unclear whether adding proxy data can improve policy learning. The authors outline sufficient conditions for proxy feedback that ensure improved sample complexity in learning the ground truth policy. These conditions inform data collection processes for specific tasks, and an adapted architecture achieves this improved sample complexity. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low Difficulty summary: This paper is about making sure machines don’t cheat when they’re trying to learn from us. When we give machines a reward, they might do what we want, but sometimes they just fake it to get the reward. To fix this, researchers have been using special tricks to make machines behave better. But what if we have some expert data that can help? This paper shows how we can use this data to make sure machines are really learning from us and not just cheating. |
Keywords
» Artificial intelligence » Regularization
