Summary of Posetalk: Text-and-audio-based Pose Control and Motion Refinement For One-shot Talking Head Generation, by Jun Ling et al.
PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation
by Jun Ling, Yiwen Wang, Han Xue, Rong Xie, Li Song
First submitted to arxiv on: 4 Sep 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI); Multimedia (cs.MM)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This study proposes PoseTalk, a system that generates lip-synchronized talking head videos with free head poses conditioned on text prompts and audio. The method uses head pose to connect visual, linguistic, and audio signals, generating motion latent from text prompts and audio cues in a pose latent space. To address the loss-imbalance problem, the authors propose a refinement-based learning strategy using two cascaded networks: CoarseNet and RefineNet. The CoarseNet estimates coarse motions, while the RefineNet focuses on learning finer lip motions by progressively estimating lip motions from low-to-high resolutions. The system outperforms state-of-the-art methods in synthesizing talking videos with natural head motions. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary PoseTalk is a new system that generates talking head videos with free head poses and lip movements that match the audio. It’s like having a real person talk to you, but instead of being a video recording, it’s generated by a computer program. The system uses text prompts and audio cues to generate motion latent from text prompts and audio cues in a pose latent space. This allows for more natural head movements and lip shapes. |
Keywords
» Artificial intelligence » Latent space
