Summary of Bonbon Alignment For Large Language Models and the Sweetness Of Best-of-n Sampling, by Lin Gui et al.
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling
by Lin Gui, Cristina Gârbacea, Victor Veitch
First submitted to arxiv on: 2 Jun 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper tackles the challenge of aligning large language models (LLMs) with human preferences using best-of-n sampling, where n samples are drawn, ranked, and the best one returned. The authors investigate two fundamental problems: first, they explore the relationship between best-of-n and approaches that train LLMs to output samples with high expected rewards. They embed both distributions in a common class of tiltings of the base LLM distribution, showing that best-of-n is essentially optimal in terms of trade-off between win-rate against the base model vs KL distance from the base model. However, best-of-n requires drawing n samples for each inference, which incurs a substantial cost. To address this, the authors derive BoNBoN Alignment to fine-tune an LLM to mimic the best-of-n sampling distribution. Experiments demonstrate that BoNBoN alignment yields significant improvements in producing a model preferred to the base policy while minimally affecting off-target aspects. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making big language models agree with what humans like. The authors want to find the best way to do this using a special technique called “best-of-n sampling”. They look at two main questions: how does best-of-n relate to other ways of training language models, and how can we make a language model mimic the best-of-n approach without having to draw lots of samples. The authors use math to show that best-of-n is actually the best way to do this trade-off between being good and not straying too far from the original model. They also create a new method called BoNBoN Alignment that makes language models behave more like what humans prefer. |
Keywords
» Artificial intelligence » Alignment » Inference » Language model
