Summary of Improving Reinforcement Learning From Human Feedback with Efficient Reward Model Ensemble, by Shun Zhang et al.
Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble
by Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, Chuang Gan
First submitted to arxiv on: 30 Jan 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper addresses a limitation in Reinforcement Learning from Human Feedback (RLHF), a widely-used approach to align large language models with human values. Current RLHF methods rely on limited human preference data to train reward models, which can lead to inaccurate predictions and misaligned outputs. To address this issue, the authors propose a novel reward ensemble method that combines multiple large language model-based reward models for more accurate predictions. The authors also explore efficient ensemble methods, including linear-layer ensemble and LoRA-based ensemble, to reduce computational complexity. Experimental results show that these ensemble methods improve the alignment performance of RLHF outputs. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps make sure that big language models are nice to humans. Right now, they’re trained on a little bit of human feedback data, but this can lead to mistakes. The researchers found a way to combine many small language model-based “reward” models into one bigger and better one. This makes the language models more likely to say what humans want them to say. The scientists tested their new method and it worked! |
Keywords
* Artificial intelligence * Alignment * Language model * Large language model * Lora * Reinforcement learning from human feedback * Rlhf
