Summary of Social Choice Should Guide Ai Alignment in Dealing with Diverse Human Feedback, by Vincent Conitzer et al.
Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback
by Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, William S. Zwicker
First submitted to arxiv on: 16 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Computer Science and Game Theory (cs.GT)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper explores the fine-tuning of foundation models like GPT-4 to avoid problematic behavior. It discusses two approaches: reinforcement learning from human feedback and constitutional AI. However, these methods may produce diverging input from humans, raising questions about aggregating and utilizing this data for collective choices. The authors argue that social choice theory can provide insights into addressing these challenges, drawing on a recent workshop on Social Choice for AI Ethics and Safety. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making sure AI models like GPT-4 don’t do bad things. It looks at two ways to fine-tune the models: getting feedback from humans or using high-level principles. But what happens when different people give different feedback? The authors think that studying how groups make decisions can help solve this problem and make collective choices about AI behavior. |
Keywords
» Artificial intelligence » Fine tuning » Gpt » Reinforcement learning from human feedback
