From Chaos to Clarity: 3DGS in the Dark

Zhihao Li, Yufei Wang, Alex Kot, Bihan Wen

Keywords: Novel view synthesis; Raw images; 3D Gaussian Splatting (3DGS); Denosing; Self-supervised learning

TL;DR: We enhance 3D Gaussian Splatting for HDR reconstruction from noisy raw images, outperforming prior models on the RawNeRF dataset.

Abstract: Novel view synthesis from raw images provides superior high dynamic range (HDR) information compared to reconstructions from low dynamic range RGB images. However, the inherent noise in unprocessed raw images compromises the accuracy of 3D scene representation. Our study reveals that 3D Gaussian Splatting (3DGS) is particularly susceptible to this noise, leading to numerous elongated Gaussian shapes that overfit the noise, thereby significantly degrading reconstruction quality and reducing inference speed, especially in scenarios with limited views. To address these issues, we introduce a novel self-supervised learning framework designed to reconstruct HDR 3DGS from a limited number of noisy raw images. This framework enhances 3DGS by integrating a noise extractor and employing a noise-robust reconstruction loss that leverages a noise distribution prior. Experimental results show that our method outperforms LDR/HDR 3DGS and previous state-of-the-art (SOTA) self-supervised and supervised pre-trained models in both reconstruction quality and inference speed on the RawNeRF dataset across a broad range of training views. We will release the code upon paper acceptance.

Primary Area: Machine vision

Submission Number: 2762