Attention-Based RGBD Fusenet for Monocular 3D Body Geometry and Pose Reconstruction

Pengle Jin, Miaopeng Li, Xinguo Liu

Abstract: This paper introduces a fully automatic method for reconstructing 3D body geometry and pose from a single RGBD image. Initially, the method combines the RGB image and the depth image to perform pose-related multi-task regression. A novel mutual-attention module is proposed, which adaptively fuses RGB and depth information while regressing multiple human features using a multi-task end-to-end deep neural network. Subsequently, the method automatically reconstructs full-body geometry and motion pose in near real-time based on these diverse features. Experimental results demonstrate that the proposed RGBD fusion network effectively extracts relevant information and eliminates ambiguous information from the two-modal inputs, significantly improving prediction accuracy in cases of appearance ambiguity, local occlusion, motion blur, and noisy input.