Summary of Posecrafter: One-shot Personalized Video Synthesis Following Flexible Pose Control, by Yong Zhong et al.
PoseCrafter: One-Shot Personalized Video Synthesis Following Flexible Pose Control
by Yong Zhong, Min Zhao, Zebin You, Xiaofeng Yu, Changwang Zhang, Chongxuan Li
First submitted to arxiv on: 23 May 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Artificial Intelligence (cs.AI)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary PoseCrafter, a one-shot method for personalized video generation following flexible poses, is introduced in this paper. Built upon Stable Diffusion and ControlNet, the approach produces high-quality videos without ground-truth frames. A reference frame from the training video is selected, inverted to initialize latent variables, and then inserted into target pose sequences with temporal attention. Latent editing through affine transformation matrices alleviates face and hand degradation. PoseCrafter outperforms baselines on several datasets under 8 metrics, following poses from different individuals or artificial edits while retaining human identity in open-domain training videos. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary PoseCrafter is a new way to make personalized videos using flexible poses. The method takes one look at the pose and makes a high-quality video without needing extra information. It uses Stable Diffusion and ControlNet to get started, then picks a reference frame from the training video to help with generation. This approach also fixes problems with faces and hands that happen when the pose is different from what’s in the training videos. PoseCrafter works well on many datasets and can even follow poses from different people or make changes to the video. |
Keywords
» Artificial intelligence » Attention » Diffusion » One shot
