Summary of Q-adapter: Customizing Pre-trained Llms to New Preferences with Forgetting Mitigation, by Yi-chen Li et al.
Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation
by Yi-Chen Li, Fuxiang Zhang, Wenjie Qiu, Lei Yuan, Chengxing Jia, Zongzhang Zhang, Yang Yu, Bo An
First submitted to arxiv on: 4 Jul 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed Q-Adapter method allows for the customization of pre-trained Large Language Models (LLMs) to new human preferences while preserving their original capabilities. This is achieved by casting LLM customization as optimizing a sum of two reward functions: one used during pre-training and another characterizing the new preference. The approach leverages the residual Q-learning framework, enabling the restoration of customized LLMs without requiring knowledge of the reward function. Experiments demonstrate the effectiveness of Q-Adapter on retaining existing knowledge and learning new preferences using the Llama-3.1 model and datasets such as DSP and HH-RLHF. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper proposes a way to make Large Language Models (LLMs) do what humans want, while still keeping their original abilities. This is done by changing how the LLMs are trained to fit new preferences. The method uses a special kind of learning called residual Q-learning and a module called an adapter. It works well on real-world datasets like DSP and HH-RLHF. |
Keywords
» Artificial intelligence » Llama » Rlhf
