Summary of Weak-to-strong Extrapolation Expedites Alignment, by Chujie Zheng et al.
Weak-to-Strong Extrapolation Expedites Alignment
by Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, Nanyun Peng
First submitted to arxiv on: 25 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed method, ExPO, leverages the initial SFT checkpoint and an already-aligned model to boost large language models’ (LLMs) alignment with human preference. By extrapolating from the weights of these two models, ExPO implicitly optimizes the alignment objective via first-order approximation. This approach bypasses additional training and data annotations, reducing costs. In experiments on HuggingFace’s twelve open-source LLMs, ExPO consistently improves off-the-shelf DPO/RLHF models, evaluated on AlpacaEval 2.0 and MT-Bench benchmarks. Scalability is demonstrated across various model sizes (1.8B to 70B) and capabilities. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary ExPO helps make language models better at following human instructions. It takes two things: an initial version of the model and a version that’s already been trained to follow human preference. ExPO uses these two versions to create an even better model without needing more training or data. This makes it faster and cheaper than before. The paper shows that ExPO works well on many different language models, no matter how big they are. |
Keywords
» Artificial intelligence » Alignment » Rlhf
