Gaussian Splatting with Neural Basis Extension

Zhi Zhou, Junke Zhu, Zhangjin Huang

Abstract: The 3D Gaussian Splatting (3D-GS) method has recently sparked a revolution in novel view synthesis with its remarkable visual effects and fast rendering speed. However, its reliance on simple spherical harmonics for color representation leads to subpar performance in complex scenes, particularly with effects like specular highlights and light refraction. Also, 3D-GS adopts a periodic split strategy, which significantly increases the model's disk space and hinders rendering efficiency. To tackle these challenges, we propose Gaussian Splatting with Neural Basis Extension (GSNB), a novel approach that substantially enhances the performance of 3D-GS in demanding scenes while reducing storage consumption. Drawing inspiration from basis function, GSNB utilizes a light-weight MLP to share feature coefficients with Spherical Harmonics (SH). This extends the color calculation of 3D Gaussians, resulting in more accurate visual effect modeling. This combination allows GSNB to achieve remarkable results even in scenes with challenging lighting and reflection conditions. Additionally, GSNB uses pre-computation to bake the MLP's output, thereby alleviating inference workload and subsequent speed loss. Furthermore, to leverage the capabilities of Neural Basis Extension and eliminate redundant Gaussians, we propose a new importance criterion to prune the converged Gaussian model and obtain a more compact representation through re-optimization. Our experimental results demonstrate that our method delivers high-quality rendering in most scenarios and effectively reduces redundant Gaussians without compromising rendering speed. The code is available at https://github.com/Dojizz/GSNB/.

External IDs:doi:10.1145/3664647.3681694