Summary of Pixel-wise Rl on Diffusion Models: Reinforcement Learning From Rich Feedback, by Mo Kordzanganeh et al.
Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback
by Mo Kordzanganeh, Danial Keshvary, Nariman Arian
First submitted to arxiv on: 5 Apr 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Computer Vision and Pattern Recognition (cs.CV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Medium Difficulty summary: Latent diffusion models have achieved state-of-the-art results in synthetic image generation. To improve these models’ alignment with human preferences, training them using reinforcement learning on human feedback is essential. Building upon denoising diffusion policy optimisation (DDPO), we introduce Pixel-wise Policy Optimisation (PXPO), a novel algorithm that incorporates pixel-level feedback to provide a more granular reward function for the model. This approach enables the model to navigate a denser reward landscape, reducing the need for large sample counts and improving its overall performance. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Low Difficulty summary: Imagine creating fake images that look super realistic. To make these images better match what humans like, we need to teach our computer models how to use feedback from people. A new way of doing this is called Pixel-wise Policy Optimisation (PXPO). It’s an improvement on a previous method and lets the model learn from smaller pieces of feedback about individual pixels in the image. This makes it easier for the model to understand what we want, without needing so many examples. |
Keywords
* Artificial intelligence * Alignment * Diffusion * Image generation * Reinforcement learning
