Summary of Float: Generative Motion Latent Flow Matching For Audio-driven Talking Portrait, by Taekyung Ki et al.
FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait
by Taekyung Ki, Dongchan Min, Gyeongsu Chae
First submitted to arxiv on: 2 Dec 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM); Image and Video Processing (eess.IV)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper introduces FLOAT, an innovative audio-driven talking portrait video generation method based on flow matching generative models. The proposed approach shifts the generative modeling from pixel-based latent spaces to learned motion latent spaces, enabling efficient design of temporally consistent motion. To achieve this, a transformer-based vector field predictor is introduced with a simple yet effective frame-wise conditioning mechanism. Additionally, FLOAT supports speech-driven emotion enhancement, allowing for natural incorporation of expressive motions. Extensive experiments demonstrate that FLOAT outperforms state-of-the-art audio-driven talking portrait methods in terms of visual quality, motion fidelity, and efficiency. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper creates a new way to make videos of people’s faces talk. It uses a special kind of computer model called a “generative model” that can create realistic videos. The model is driven by audio, so it can make the video match what someone is saying. This is important because it allows for more natural and expressive motions in the video. The paper also shows how to add emotions to the video based on the audio, which makes it look even more real. Overall, this new method creates higher-quality videos that are more realistic and engaging. |
Keywords
» Artificial intelligence » Generative model » Transformer
