Summary of Rlhf From Heterogeneous Feedback Via Personalization and Preference Aggregation, by Chanwoo Park et al.
RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
by Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, Asuman Ozdaglar
First submitted to arxiv on: 30 Apr 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary A reinforcement learning framework for aligning AI systems with human values is proposed to address issues of heterogeneous human preferences and strategic behavior in providing feedback. The paper presents two frameworks: personalization-based and aggregation-based approaches. Personalization-based methods use representation learning and clustering to learn multiple reward models, trading off bias and variance. Sample complexity guarantees are established for these approaches. Aggregation-based methods aggregate diverse and truthful preferences from humans using utilitarianism and Leximin approaches or probabilistic opinions. The paper also develops a mechanism design approach to handle strategic human labelers who may manipulate aggregated preferences with untruthful feedback. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Reinforcement learning is a way to train AI systems to make good decisions by giving them rewards for good choices. But people have different ideas about what’s good, and some might try to trick the system into making bad choices. This paper proposes two new ways to handle these issues: one that learns individual preferences and another that combines all the opinions together. The first approach uses special math techniques to learn multiple models of good behavior, while the second approach averages out all the different opinions to get a single idea of what’s best. |
Keywords
» Artificial intelligence » Clustering » Reinforcement learning » Representation learning
