Summary of Disentangling Foreground and Background Motion For Enhanced Realism in Human Video Generation, by Jinlin Liu et al.
Disentangling Foreground and Background Motion for Enhanced Realism in Human Video Generation
by Jinlin Liu, Kai Yu, Mengyang Feng, Xiefan Guo, Miaomiao Cui
First submitted to arxiv on: 26 May 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Recent advancements in human video synthesis have enabled the generation of high-quality videos through stable diffusion models. Existing methods primarily focus on animating the human element (foreground) guided by pose information, while neglecting dynamic backgrounds. Our technique simultaneously learns foreground and background dynamics using distinct motion representations. Foreground figures are animated leveraging pose-based motion, capturing intricate actions, whereas sparse tracking points model background motion, reflecting natural interactions between foreground activity and environmental changes. Training on real-world videos with our innovative motion depiction approach, our model generates videos exhibiting coherent movement in both foreground subjects and surrounding contexts. To extend video generation to longer sequences without accumulating errors, we adopt a clip-by-clip strategy, introducing global features at each step. We link the final frame of a produced clip with input noise to spawn the succeeding one, maintaining narrative flow. Empirical evaluations demonstrate our method’s superiority in producing videos with harmonious interplay between foreground actions and responsive background dynamics, surpassing prior methodologies. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary Imagine watching movies or TV shows where characters move around in real-life environments that change too! That’s what this new technology does – it makes backgrounds move along with the people. Right now, most videos only animate the people (called foreground) and leave the background static. But our new method learns how both the people and background should move together. We use special ways to track movement in both areas and train our computer model on lots of real-life video examples. Then, we can make longer videos without mistakes by breaking it down into smaller chunks and linking each part to the next one smoothly. Our results show that this new technology creates videos with people moving naturally alongside changing backgrounds, doing better than earlier methods. |
Keywords
» Artificial intelligence » Diffusion » Tracking
