Summary of Laser: Learning to Adaptively Select Reward Models with Multi-armed Bandits, by Duy Nguyen et al.
LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits
by Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal
First submitted to arxiv on: 2 Oct 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces LASeR (Learning to Adaptively Select Rewards), a novel approach for iteratively training Large Language Models (LLMs) using multiple Reward Models (RMs). The authors argue that fixed RMs can be suboptimal, and optimizing LLMs with multiple RMs simultaneously is computationally-intensive. LASeR frames this problem as a multi-armed bandit problem, selecting the most well-suited RM for each instance to rank outputs and generate preference data. The approach boosts iterative LLM optimization, improving absolute average accuracy by 2.67% on three datasets and achieving a 71.45% win rate on WildChat. LASeR also generalizes to long-context generation tasks, with an average improvement of 2.64 F1 and 2.42 F1 on single- and multi-document QA. The method is robust to noisy rewards and adapts to multiple settings. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary LASeR is a new way to train language models using many different “reward” systems at the same time. This helps the model learn better by choosing the best system for each task. The results show that this approach works well, with an improvement of 2.67% on three datasets and a 71.45% success rate on another test. LASeR also does well on longer text generation tasks. It’s good at handling noisy rewards and changing its approach depending on the task. |
Keywords
» Artificial intelligence » Optimization » Text generation
