OccFusion: Rendering Occluded Humans with Generative Diffusion Priors

Adam Sun, Tiange Xiang, Scott Delp, Li Fei-Fei, Ehsan Adeli

Keywords: vision, gaussian splatting, human, rendering, diffusion

TL;DR: We reconstruct humans from occluded monocular videos with Gaussian splatting assisted by generative diffusion priors.

Abstract: Most existing human rendering methods require every part of the human to be fully visible throughout the input video. However, this assumption does not hold in real-life settings where obstructions are common, resulting in only partial visibility of the human. Considering this, we present OccFusion, an approach that utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion models for efficient and high-fidelity human rendering. We propose a pipeline consisting of three stages. In the Initialization stage, complete human masks are generated from partial visibility masks. In the Optimization stage, 3D human Gaussians are optimized with additional supervision by Score-Distillation Sampling (SDS) to create a complete geometry of the human. Finally, in the Refinement stage, in-context inpainting is designed to further improve rendering quality on the less observed human body parts. We evaluate OccFusion on ZJU-MoCap and challenging OcMotion sequences and find that it achieves state-of-the-art performance in the rendering of occluded humans

Submission Number: 28