Real-Time Human Motion Transfer System for Holographic Displays

Wenkang Shan, Haopeng Lu, Chuanmin Jia, Xinfeng Zhang, Siwei Ma, Yaqiang Wu, Wen Gao

Abstract: This paper presents a human motion transfer system designed for holographic displays. It utilizes a single RGB camera to capture real-time human movement data. The captured data is then processed through off-the-shelf models for 2D and 3D human pose estimation. Subsequently, the estimated poses are retargeted to a pre-defined 3D avatar using forward kinematics, enabling high-fidelity motion transfer. Demonstrations on both a holographic cabin and Microsoft HoloLens 2 showcase the system's ability to enhance the expressiveness and authenticity of digital humans, thus paving the way for more intuitive interactions within the metaverse.