Summary of Reward Fine-tuning Two-step Diffusion Models Via Learning Differentiable Latent-space Surrogate Reward, by Zhiwei Jia et al.
Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward
by Zhiwei Jia, Yuesong Nan, Huixi Zhao, Gengdai Liu
First submitted to arxiv on: 22 Nov 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Medium Difficulty summary: Recent research has demonstrated the feasibility of fine-tuning diffusion models (DMs) with arbitrary rewards, including non-differentiable ones, using reinforcement learning (RL) techniques. However, applying existing RL methods to step-distilled DMs for ultra-fast image generation is challenging. Our analysis highlights limitations of policy-based RL methods like PPO or DPO towards this goal. To address these limitations, we propose fine-tuning DMs with learned differentiable surrogate rewards, named LaSRO. LaSRO learns surrogate reward models in the latent space of SDXL to convert arbitrary rewards into differentiable ones for effective reward gradient guidance. Our method leverages pre-trained latent DMs for reward modeling and tailors reward optimization for ultra-fast image generation with efficient off-policy exploration. We show that LaSRO outperforms popular RL methods, including DDPO and Diffusion-DPO, in improving ultra-fast image generation with different reward objectives. Furthermore, we demonstrate LaSRO’s connection to value-based RL, providing theoretical insights. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low Difficulty summary: This paper is about making computers generate images really fast using a special kind of artificial intelligence called diffusion models. The problem is that these models can’t understand what they’re supposed to be generating if the instructions are tricky or non-mathematical. The researchers found that existing ways of training these models didn’t work well for this task. So, they came up with a new approach that uses something called “surrogate rewards” to help the model learn faster and better. They tested their method, called LaSRO, and showed that it’s really good at generating images quickly and efficiently. |
Keywords
» Artificial intelligence » Diffusion » Fine tuning » Image generation » Latent space » Optimization » Reinforcement learning
