Summary of Gaussianspeech: Audio-driven Gaussian Avatars, by Shivangi Aneja et al.
GaussianSpeech: Audio-Driven Gaussian Avatars
by Shivangi Aneja, Artem Sevastopolsky, Tobias Kirschstein, Justus Thies, Angela Dai, Matthias Nießner
First submitted to arxiv on: 27 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Graphics (cs.GR); Sound (cs.SD); Audio and Speech Processing (eess.AS)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary The proposed GaussianSpeech approach synthesizes high-fidelity animation sequences of personalized 3D human head avatars from spoken audio. This is achieved by coupling speech signals with 3D Gaussian splatting to capture expressive, detailed facial movements. A compact avatar representation generates expression-dependent color and leverages losses for wrinkle and perceptual detail. An audio-conditioned transformer model extracts lip and expression features from audio input. To train the approach, a new large-scale dataset of talking humans was captured with diverse facial geometry. GaussianSpeech achieves state-of-the-art performance at real-time rendering rates, encompassing various expressions and styles. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary GaussianSpeech is a new way to create realistic animations of people speaking. It takes spoken audio as input and generates high-quality 3D animations of human heads. The approach uses special techniques like Gaussian splatting and transformer models to capture the details of facial movements, skin wrinkles, and expression changes. To train the system, researchers created a large dataset of videos of people speaking from different accents and with diverse facial features. The result is an animation that looks natural and can be generated in real-time. |
Keywords
» Artificial intelligence » Transformer
