Joint Geometry–Appearance Human Reconstruction in a Unified Latent Space via Bridge Diffusion

Yingzhi Tang, Qijian Zhang, Junhui Hou

Keywords: 3D human reconstruction, geometry, diffusion

Abstract:

Reconstructing both geometry and appearance of a digital human from a single image remains highly challenging. Existing approaches typically decouple geometry and appearance, employing separate models for each, which limits their ability to reconstruct digital humans in a unified manner.
In this paper, we propose JGA-LBD, which formulates human reconstruction as a bridge diffusion task in a unified latent space, yielding a joint latent representation that encodes both geometry and appearance. We address the challenge of human reconstruction from heterogeneous conditions, i.e., depth maps and SMPL models estimated from RGB images. Directly combining heterogeneous modalities introduces substantial training difficulties, to overcome this, we unify all conditions into 3D Gaussian representation and compress them into a unified latent space using a sparse variantional autoencoder. All diffusion learning is then conducted within this unified latent space, which markedly reduces optimization complexity. Our setting strikingly lends itself to bridge diffusion: the depth map can be regarded as a partial observation of the target latent code, enabling the model to focus solely on inferring the missing components. Finally, a decoding module reconstructs geometry and renders novel-view images from the latent representation. Experiments demonstrate that JGA-LBD outperforms state-of-the-art methods in both geometry and appearance, and generates plausible results on in-the-wild images.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 4413