Visibility-guided Human Body Reconstruction from Uncalibrated Multi-view Cameras

Zhenyu Xie, Huanyu He, Gui Zou, Jie Wu, Guoliang Liu, Jun Zhao, Yingxue Wang, Hui Lin, Weiyao Lin

Abstract: We present a novel method for 3D human body reconstruction with multi-view images from calibration-free cameras by multi-view fusion with explicit visibility modelling. Existing multi-view methods usually establish geometric constraints by using accurate camera intrinsic and extrinsic parameters. Despite remarkable performances, multi-view camera calibration often requires complex operations and additional maintenance to fix camera positions and angles, which restrict its applicability to real-world scenarios. In contrast, we leverage vertex-wise visibility prediction as calibration cues to guide the multi-view human body aggregation, which eliminates the need for camera calibration. Specifically, we estimate the UV position map and the vertex-wise visibility map of human body in each camera view, which allows us to align and aggregate multi-view information in a hierarchical manner. To further improve the alignment between human body and vertex-wise visual features, we propose an Occlusion-aware UV-pixel Refinement (OUVR) module, which takes the previous result of coarse alignment as input. The visible vertices are disentangled from the UV map and are reprojected on the image to describe the misalignment of current body estimation and image features. The UV map representation is adopted throughout the refinement process to avoid the potential error propagation brought by parametric representation. The effectiveness of our approach is validated on 3D human body reconstruction, as it surpasses current leading multi-view fusion methods, and showing comparable performance to methods that require accurate multi-view camera calibration.