3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with View-consistent 2D Diffusion Priors

Xi Liu, Chaoyi Zhou, Siyu Huang

Keywords: 3D model enhancement, 3D Guassian splatting, novel view synthesis, diffusion model, image restoration

TL;DR: We propose a method that exploits view-consistent 2D generative priors, i.e., a video diffusion model, to enhance 3D Gaussian splatting rendering quality.

Abstract:

Novel-view synthesis aims to generate novel views of a scene from multiple input
images or videos, and recent advancements like 3D Gaussian splatting (3DGS)
have achieved notable success in producing photorealistic renderings with efficient
pipelines. However, generating high-quality novel views under challenging settings,
such as sparse input views, remains difficult due to insufficient information in
under-sampled areas, often resulting in noticeable artifacts. This paper presents
3DGS-Enhancer, a novel pipeline for enhancing the representation quality of
3DGS representations. We leverage 2D video diffusion priors to address the
challenging 3D view consistency problem, reformulating it as achieving temporal
consistency within a video generation process. 3DGS-Enhancer restores view-
consistent latent features of rendered novel views and integrates them with the
input views through a spatial-temporal decoder. The enhanced views are then
used to fine-tune the initial 3DGS model, significantly improving its rendering
performance. Extensive experiments on large-scale datasets of unbounded scenes
demonstrate that 3DGS-Enhancer yields superior reconstruction performance and
high-fidelity rendering results compared to state-of-the-art methods. The project
webpage is https://xiliu8006.github.io/3DGS-Enhancer-project.

Primary Area: Machine vision

Submission Number: 4219