Sp2360: Sparse-view 360◦ Scene Reconstruction using Cascaded 2D Diffusion Priors

Soumava Paul, Christopher Wewer, Bernt Schiele, Jan Eric Lenssen

Keywords: sparse-view 3D reconstruction, 2D diffusion priors

TL;DR: Synthesizing novel views with a combination of diffusion-based image inpainting and artifact removal priors helps reconstruct complex 360° scenes without 3D-aware finetuning of a 2D diffusion model on million-scale multiview datasets.

Abstract: We aim to tackle sparse-view reconstruction of a $360^{\circ}$ 3D scene using priors from latent diffusion models (LDM). The sparse-view setting is ill-posed and underconstrained, especially for scenes where the camera rotates $360$ degrees around a point, as no visual information is available beyond some frontal views focused on the central object(s) of interest. In this work, we show that pretrained 2D diffusion models can strongly improve the reconstruction of a scene with low-cost fine-tuning, alleviating reliance on large-scale 3D datasets. Specifically, we present SparseSplat360 (Sp$^2$360), a method that employs a cascade of in-painting and artifact removal models to fill in missing details and clean novel views. Due to superior training and rendering speeds, we use an explicit scene representation in the form of 3D Gaussians over NeRF-based implicit representations. We propose an iterative update strategy to fuse generated pseudo novel views with existing 3D Gaussians fitted to the initial sparse inputs. As a result, we obtain a multi-view consistent scene representation with details coherent with the observed inputs. Our evaluation on the challenging Mip-NeRF360 dataset shows that our proposed 2D to 3D distillation algorithm considerably improves the performance of a regularized version of 3DGS adapted to a sparse-view setting and outperforms existing sparse-view reconstruction methods in $360^{\circ}$ scene reconstruction on traditional metrics. Qualitatively, our method generates entire $360^{\circ}$ scenes from as few as $9$ input views, with a high degree of foreground and background detail.

Submission Number: 41