Summary of Diffrelight: Diffusion-based Facial Performance Relighting, by Mingming He et al.
DifFRelight: Diffusion-Based Facial Performance Relighting
by Mingming He, Pascal Clausen, Ahmet Levent Taşel, Li Ma, Oliver Pilarski, Wenqi Xian, Laszlo Rikker, Xueming Yu, Ryan Burgert, Ning Yu, Paul Debevec
First submitted to arxiv on: 10 Oct 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Graphics (cs.GR)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. Our framework leverages a subject-specific dataset containing diverse facial expressions captured under various lighting conditions to train a diffusion model for precise lighting control. This model enables high-fidelity relit facial images from flat-lit inputs, using spatially-aligned conditioning of flat-lit captures and random noise, along with integrated lighting information for global control. We apply this model to dynamic facial performances captured in a consistent flat-lit environment and reconstruct for novel-view synthesis using a scalable dynamic 3D Gaussian Splatting method. Our framework also enables high dynamic range imaging (HDRI) composition using multiple directional lights to produce dynamic sequences under complex lighting conditions. Evaluations demonstrate the models efficiency in achieving precise lighting control and generalizing across various facial expressions, preserving detailed features like skintexture and hair. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary We’ve developed a new way to make faces look more realistic in videos. Our method uses computers to adjust the lighting on a face so it looks like the person is being lit from different directions. We trained our computer model using lots of pictures of people’s faces with different expressions and lighting conditions. Then, we used this model to change the lighting on videos of people talking or performing. The results are very realistic and look like they were filmed in different lighting conditions. Our method can even make eye reflections and other details look more natural. This technology could be useful for making movies or TV shows that have complex lighting effects. |
Keywords
» Artificial intelligence » Diffusion » Diffusion model » Translation
