Summary of Prdp: Proximal Reward Difference Prediction For Large-scale Reward Finetuning Of Diffusion Models, by Fei Deng et al.
PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models
by Fei Deng, Qifei Wang, Wei Wei, Matthias Grundmann, Tingbo Hou
First submitted to arxiv on: 13 Feb 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper proposes Proximal Reward Difference Prediction (PRDP), a new approach to aligning foundation models with downstream objectives in the vision domain. By using a supervised regression objective that predicts the reward difference of generated image pairs, PRDP enables stable black-box reward finetuning for diffusion models on large-scale prompt datasets. This is achieved by leveraging the relationship between the RL and RDP objectives, which ensures that the optimal solution for both is the same. The authors demonstrate that PRDP can match the performance of established RL-based methods in small-scale training and outperforms them in large-scale training. Specifically, PRDP achieves superior generation quality on complex, unseen prompts from datasets such as Human Preference Dataset v2 and Pick-a-Pic v1. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper helps us make better computers that can create images like humans do. It’s called “reward finetuning” because we want the computer to learn what makes an image good or bad. The authors invented a new way to do this using something called “proximal reward difference prediction.” This lets them teach the computer to predict how good or bad an image is by comparing it to another version of the same image that’s a little different. They tested their method on many images and found that it worked much better than other methods. |
Keywords
* Artificial intelligence * Prompt * Regression * Supervised
