Summary of Brain: Bayesian Reward-conditioned Amortized Inference For Natural Language Generation From Feedback, by Gaurav Pandey et al.
BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback
by Gaurav Pandey, Yatin Nandwani, Tahira Naseem, Mayank Mishra, Guangxuan Xu, Dinesh Raghu, Sachindra Joshi, Asim Munawar, Ramón Fernandez Astudillo
First submitted to arxiv on: 4 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed paper addresses the underperformance of distribution matching methods for language model alignment, such as Generation with Distributional Control (GDC) and Distributional Policy Gradient (DPG), in reinforcement learning from human feedback (RLHF). The main issue identified is high variance of the gradient estimate, which leads to a lack of success. To mitigate this, the authors introduce a self-normalized baseline to reduce variance. Additionally, they generalize target distributions using Bayes’ rule and propose BRAIn – Bayesian Reward-conditioned Amortized Inference, which serves as a bridge between distribution matching methods and Direct Preference Optimization (DPO). The results demonstrate significant improvement in summarization and Antropic HH tasks. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper explores ways to improve language model alignment by addressing the lack of attention given to distribution matching methods. It finds that high variance in gradient estimates is the main reason for their underperformance. To fix this, it proposes a new approach called BRAIn, which uses Bayes’ rule to define reward-conditioned posteriors. This approach outperforms previous methods in summarization and Antropic HH tasks. |
Keywords
* Artificial intelligence * Alignment * Attention * Inference * Language model * Optimization * Reinforcement learning from human feedback * Rlhf * Summarization
