Summary of Copr: Continual Human Preference Learning Via Optimal Policy Regularization, by Han Zhang et al.
COPR: Continual Human Preference Learning via Optimal Policy Regularization
by Han Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai, Yehong Zhang, Yulan He, Hui Wang, Yue Yu, Kam-Fai Wong, Bin Liang, Ruifeng Xu
First submitted to arxiv on: 22 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes a novel method called Continual Optimal Policy Regularization (COPR) to align Large Language Models with human preferences in a continual learning setting. The authors address the challenges of Catastrophic Forgetting and unbalanced objectives by utilizing Lagrangian Duality and sampling distributions as regularization constraints. They demonstrate the effectiveness of COPR through experiments on a proposed benchmark, outperforming strong CL baselines in terms of reward-based and human evaluations. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps Large Language Models learn from humans better. It’s like teaching a machine to understand what we want it to do, but instead of stopping once it gets it right, the machine keeps learning and adapting to our changing preferences. The authors created a new way called COPR that makes sure the machine doesn’t forget what it learned before and also doesn’t get too focused on one thing at the expense of others. They tested this method and showed that it works better than other ways they tried. |
Keywords
* Artificial intelligence * Continual learning * Regularization
