Self-Supervised Human Mesh Recovery from Partial Point Cloud via a Self-Improving Loop

Chang Su, Beihong Jin, Fusang Zhang, Siheng Li, Zhi Wang

Abstract: Accurate 3D human mesh recovery from point clouds remains challenging. Most existing methods depend on full 3D supervision or complete input data, both of which are difficult to obtain in practice. This calls for robust solutions to handle partial point clouds in a self-supervised manner. To tackle the dual challenges of point cloud incompleteness and the absence of supervision signals, we propose a novel method named \textbf{SS-HMR}, which offers three key insights. First, we estimate point-wise semantics in a self-supervised manner to match partial inputs with a canonical template. The resulting correspondences serve as supervision signals for the regression network in human mesh recovery. Second, we incorporate regression-based and optimization-based paradigms into a self-improving loop: the regression network provides strong initialization for optimization, while the optimization routine generates pseudo-labels that, in turn, enhance the regression network. This mutual feedback enables more accurate and stable mesh recovery over time. Third, generating multiple initializations and selecting the best result mitigates the optimization routine’s sensitivity to initialization, improving robustness to sparse and noisy data. Extensive experiments are conducted on three public datasets and results demonstrate that SS-HMR outperforms existing methods. Notably, SS-HMR performs excellently on different test data, whether from original point clouds captured by depth cameras or LiDAR devices, or from noise-added ones. This shows that SS-HMR has strong generalization ability and robustness across different data sources. Codes are available at \url{https://github.com/suchang-99/SS-HMR}.