Summary of Value Augmented Sampling For Language Model Alignment and Personalization, by Seungwook Han et al.
Value Augmented Sampling for Language Model Alignment and Personalization
by Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, Pulkit Agrawal
First submitted to arxiv on: 10 May 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The paper presents a new framework called Value Augmented Sampling (VAS) that enables aligning Large Language Models (LLMs) to different human preferences, learning new skills, and unlearning harmful behavior. VAS optimizes reward functions using data sampled from an initial, frozen LLM, solving the optimization challenges in co-training value function and policy. The framework outperforms established baselines like PPO and DPO on standard benchmarks and achieves comparable results to Best-of-128 with lower inference cost. Unlike existing RL methods, VAS does not require access to the weights of the pre-trained LLM. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper creates a new way to make big language models work better for people. It’s like teaching an AI to behave nicely and learn new things, but without making it too complicated or changing what it already knows. The new method, called Value Augmented Sampling (VAS), is faster and works better than other methods that try to do the same thing. |
Keywords
» Artificial intelligence » Inference » Optimization
