Summary of Generalized Preference Optimization: a Unified Approach to Offline Alignment, by Yunhao Tang et al.
Generalized Preference Optimization: A Unified Approach to Offline Alignment
by Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, Bilal Piot
First submitted to arxiv on: 8 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Offline preference optimization has shown promise in recent alignment practices. Our proposed generalized preference optimization (GPO) framework unifies existing algorithms like DPO, IPO, and SLiC as special cases while introducing new variants. GPO parameterizes offline losses with convex functions, shedding light on how offline algorithms enforce regularization through the design of these functions. Analyzing and experimenting with GPO reveals connections and subtle differences between offline regularization and KL divergence regularization from RLHF. In a controlled setting similar to Gao et al 2023, we demonstrate that different GPO variants achieve similar trade-offs between regularization and performance, though optimal hyperparameter values may differ as predicted by theory. This work provides new algorithmic toolkits and empirical insights for alignment practitioners. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine you want to fine-tune a big model using data from the past. This can be tricky because it’s hard to get right. Our team has come up with a way to make this process better by unifying different methods into one framework called GPO (Generalized Preference Optimization). GPO helps us understand how these methods work and how they compare to each other. We found that all these methods aim for the same goal – finding a good balance between making sure the model doesn’t get too wild and performing well. Our results show that different versions of this framework can achieve similar results, but with slightly different settings. |
Keywords
* Artificial intelligence * Alignment * Hyperparameter * Optimization * Regularization * Rlhf
