Physical Inertial Poser (PIP)

Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors

Xinyu Yi1     Yuxiao Zhou1     Marc Habermann2     Soshi Shimada2     Vladislav Golyanik2     Christian Theobalt2     Feng Xu1
1Tsinghua University       2Max Planck Institute for Informatics, Saarland Informatics Campus


Accepted by CVPR 2022 (Best Paper Finalist)

PIP captures physically correct human motion, joint torques, and ground reaction forces solely from 6 IMUs.
Importantly, PIP runs at 60 fps with only 16ms latency, enabling real-time applications such as gaming.

Abstract

Motion capture from sparse inertial sensors has shown great potential compared to image-based approaches since occlusions do not lead to a reduced tracking quality and the recording space is not restricted to be within the viewing frustum of the camera. However, capturing the motion and global position only from a sparse set of inertial sensors is inherently ambiguous and challenging. In consequence, recent state-of-the-art methods can barely handle very long period motions, and unrealistic artifacts are common due to the unawareness of physical constraints. To this end, we present the first method which combines a neural kinematics estimator and a physics-aware motion optimizer to track body motions with only 6 inertial sensors. The kinematics module first regresses the motion status as a reference, and then the physics module refines the motion to satisfy the physical constraints. Experiments demonstrate a clear improvement over the state of the art in terms of capture accuracy, temporal stability, and physical correctness.


Video


More Live Captures

Comparisons with TransPose


Acknowledgments

This work was supported by Beijing Natural Science Foundation (JQ19015), the NSFC (No.61727808, 62021002), the National Key R&D Program of China 2018YFA0704000. This work was supported by THUIBCS, Tsinghua University and BLBCI, Beijing Municipal Education Commission. This work was partially supported by the ERC consolidator grant 4DReply (770784). We thank Notiom for the extensive support on inertial sensors, and Liuqing Yang, Liangdi Ma, Siyuan Teng, Wenbin Lin for the help on live demos. Feng Xu is the corresponding author.


Citation

@InProceedings{PIPCVPR2022,
  author = {Yi, Xinyu and Zhou, Yuxiao and Habermann, Marc and Shimada, Soshi and Golyanik, Vladislav and Theobalt, Christian and Xu, Feng},
  title = {Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2022}
}