GenCoGS: Generative Completion-based 3D Gaussian Splatting for High-Fidelity Few-Shot Novel View Synthesis

Junao Shen, Haojie Dong, Tian Feng, Ruihong Ye, Wenjie Wang, Yuandong Zhang, Tianjia Shao

Keywords: 3D Gaussians Splatting, Novel View Synthesis, few-shot, Generative Model

Abstract:

Conventional few-shot novel view synthesis (NVS) methods using 3D Gaussian Splatting (3DGS) have demonstrated significance, but remain constrained by their overdependence on the limited information from training views. Their unsatisfactory scene completion capability would underrepresent those scene regions either unobserved in training views or with local details and thus cause floating artifacts against high fidelity.
To address these challenges, we propose GenCoGS, a unified 3DGS-based few-shot NVS method focusing on initializing and optimizing 3DGS representation using generative completion-based strategies to enhance scene completion.
Specifically, our generative point cloud completion-based strategy produces and filters complementary points toward a complete point cloud with refined structural and appearance information for Gaussian initialization;
The generative pseudo view completion-based strategy leverages an image-to-video diffusion model to synthesize complete pseudo views, which benefits Gaussian optimization especially within unobserved scene regions and mitigates hallucination for less appearance distortion.
Integrating both strategies enables accurate and coherent scene completion for high-fidelity few-shot NVS.
Extensive experiments on three benchmark datasets demonstrate the superiority of our GenCoGS for few-shot NVS evaluated in common metrics compared to baseline methods. Compared to those 3DGS-based few-shot NVS methods, our GenCoGS achieves improvements of up to $2.40$dB, $0.08$ and $0.125$ in PSNR, SSIM and LPIPS.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 5994