Summary of Dense Reward For Free in Reinforcement Learning From Human Feedback, by Alex J. Chan et al.
Dense Reward for Free in Reinforcement Learning from Human Feedback
by Alex J. Chan, Hao Sun, Samuel Holt, Mihaela van der Schaar
First submitted to arxiv on: 1 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a significant improvement to Reinforcement Learning from Human Feedback (RLHF), which has enabled Large Language Models (LLMs) to effectively follow instructions. Traditionally, RLHF involves generating completions from the LLM and then using a separate reward model to assign a score. However, this process is challenging due to the sparse rewards provided by the auto-regressive process. To address this issue, the authors leverage the attention weights calculated by the transformer architecture in the reward model. These attention weights are used to redistribute the reward along the completion, providing a denser signal that highlights important tokens. This approach is theoretically equivalent to potential-based reward shaping and empirically shows improved training stability, learning rates, and local optima. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps make language models better at following instructions by changing how they get feedback. Right now, these models are trained using rewards that tell them whether their answers are correct or not. But this process is hard because the model only gets one reward at the end of a task. To fix this, the authors look at the attention weights calculated while training the model and use those to give more feedback along the way. This makes it easier for the model to learn and improves its performance. |
Keywords
* Artificial intelligence * Attention * Reinforcement learning from human feedback * Rlhf * Transformer
