Summary of Xr-mbt: Multi-modal Full Body Tracking For Xr Through Self-supervision with Learned Depth Point Cloud Registration, by Denys Rozumnyi et al.
XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration
by Denys Rozumnyi, Nadine Bertsch, Othman Sbai, Filippo Arcadu, Yuhua Chen, Artsiom Sanakoyeu, Manoj Kumar, Catherine Herold, Robin Kips
First submitted to arxiv on: 27 Nov 2024
Categories
- Main: Computer Vision and Pattern Recognition (cs.CV)
- Secondary: Machine Learning (cs.LG)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary This paper presents a novel approach to tracking full-body motions in extended reality (XR) devices, particularly AR/VR applications. The authors aim to overcome the limitations of current methods that rely on 3-point signals from head and controller tracking. They propose leveraging depth sensing information from XR devices, combined with self-supervision, to learn a multi-modal pose estimation model. This approach enables real-time tracking of full-body motions in XR environments. The proposed method extends existing 3-point motion synthesis models to point cloud modalities using a semantic point cloud encoder network and a residual network for multi-modal pose estimation. These modules are trained jointly in a self-supervised manner, utilizing a combination of real unregistered point clouds and simulated data from motion capture. Compared to state-of-the-art systems, the proposed method accurately tracks various body motions. The authors demonstrate that XR-MBT can track legs in XR for the first time, whereas traditional synthesis approaches based on partial body tracking are blind. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making virtual reality (VR) and augmented reality (AR) devices feel more real by letting them track our whole bodies. Right now, VR/AR devices only track head movements and a few hand or arm movements. To make these experiences feel more natural, we need to be able to track the rest of our body too. The problem is that current devices don’t have sensors to track our legs and other parts of our body. This paper proposes a new way to use the sensors that VR/AR devices already have to track our whole body in real-time. It’s like teaching a computer to recognize what our body is doing without needing any special equipment. |
Keywords
» Artificial intelligence » Encoder » Multi modal » Pose estimation » Residual network » Self supervised » Tracking
