Summary of Reward Difference Optimization For Sample Reweighting in Offline Rlhf, by Shiqi Wang et al.
Reward Difference Optimization For Sample Reweighting In Offline RLHF
by Shiqi Wang, Zhengze Zhang, Rui Zhao, Fei Tan, Cam Tu Nguyen
First submitted to arxiv on: 18 Aug 2024
Categories
- Main: Computation and Language (cs.CL)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The abstract proposes a solution to align Large Language Models (LLMs) with human preferences. The authors recognize that Reinforcement Learning with Human Feedback (RLHF) is effective but resource-intensive, so they introduce offline RLHF as an alternative. Offline RLHF directly optimizes LLMs with ranking losses on a fixed preference dataset. However, current methods only capture the ordinal relationship between responses, overlooking how much one response is preferred over another. To address this issue, the authors propose Reward Difference Optimization (RDO), which uses reward difference coefficients to reweigh sample pairs in offline RLHF. They also develop a difference model that captures rich interactions between response pairs for predicting these coefficients. The method is tested on 7B LLMs using automatic metrics and human evaluation, with promising results. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary The paper proposes a new way to make Large Language Models (LLMs) match what humans like or dislike. Right now, there are ways to train LLMs with human feedback, but they can be tricky and take a lot of resources. The authors suggest an easier method called offline RLHF that directly trains the model using a fixed dataset. However, this method only looks at how responses compare to each other, not how much one is preferred over another. To fix this, the authors introduce Reward Difference Optimization (RDO) that uses special coefficients to make the model better understand how humans prefer certain responses. The results look good so far. |
Keywords
» Artificial intelligence » Optimization » Reinforcement learning » Rlhf
