NeuSD: Surface Completion With Multi-View Text-to-Image Diffusion

Savva Ignatyev, Daniil Selikhanovych, Oleg Voynov, Yiqun Wang, Peter Wonka, Stamatios Lefkimmiatis, Evgeny Burnaev

Abstract: We present a new method for 3D surface reconstruction from multiple images where only a part of the object of interest is captured. Our approach builds on two recent developments: surface reconstruction with neural radiance fields for reconstruction of the visible parts of the surface, and the guidance of pre-trained 2D diffusion models in the form of Score Distillation Sampling (SDS) for plausible synthesis of the unobserved parts. We obtain geometric details on the synthesized part by employing normal maps as a pure geometric representation for SDS instead of color renderings which are entangled with the appearance information. We propose a way to reduce the stochasticity of the SDS loss which results in an increased level of detail of the surface. Finally, we propose a tuning-free mechanism to achieve multi-view consistency in SDS, which increases the consistency between the synthesized and visible parts of the surface. We evaluate our approach on the BlendedMVS dataset and demonstrate significant qualitative and quantitative improvements over competing methods.