Gaussian Splatting With Neural Basis Extension

Zhi Zhou, Junke Zhu, ZhangJin Huang

Abstract: The 3D Gaussian Splatting(3D-GS) method has recently sparked a new revolution in novel view synthesis with its remarkable visual effects and fast rendering speed. However, its reliance on simple spherical harmonics for color representation leads to subpar performance in complex scenes, struggling with effects like specular highlights, light refraction, etc. Also, 3D-GS adopts a periodic split strategy, significantly increasing the model's disk space and hindering rendering efficiency. To tackle these challenges, we introduce Gaussian Splatting with Neural Basis Extension (GSNB), a novel approach that substantially improves the performance of 3D-GS in demanding scenes while reducing storage consumption. Drawing inspiration from basis function, GSNB employs a light-weight MLP to share feature coefficients with spherical harmonics and extends the color calculation of 3D Gaussians for more precise visual effect modeling. This combination enables GSNB to achieve impressive results in scenes with challenging lighting and reflection conditions. Moreover, GSNB utilizes pre-computation to bake the network's output, thereby alleviating inference workload and subsequent speed loss. Furthermore, to leverage the capabilities of Neural Basis Extension and eliminate redundant Gaussians, we propose a new importance criterion to prune the converged Gaussian model and obtain a more compact representation through re-optimization. Experimental results demonstrate that our method delivers high-quality rendering in most scenarios and effectively reduces redundant Gaussians without compromising rendering speed. Our code and real-time demos will be released soon.

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Content] Media Interpretation

Relevance To Conference: Our work, Gaussian Splatting with Neural Basis Extension (GSNB), has great potential to improve the user experience in multimedia applications. Given a set of images, GSNB achieves high-quality real-time rendering of indoor or outdoor scenes and is able to accurately render intricate visual effects, including specular highlights and light refraction. This capability promises to enhance the user experience in various multimedia platforms that require high visual fidelity, such as virtual reality, augmented reality, 3D scene visualization, digital art exhibitions or interactive storytelling. The improved visual representation enabled by GSNB provides a solid foundation for the display of these multimedia content.

Submission Number: 5663