SpecGaussian with latent features: A high-quality modeling of the view-dependent appearance for 3D Gaussian Splatting

Zhiru Wang, Shiyun Xie, Chengwei Pan, Guoping Wang

Abstract: Recently, the 3D Gaussian Splatting (3D-GS) method has achieved great success in novel view synthesis, providing real-time rendering while ensuring high-quality rendering results. However, this method faces challenges in modeling specular reflections and handling anisotropic appearance components, especially in dealing with view-dependent color under complex lighting conditions. Additionally, 3D-GS uses spherical harmonic to learn the color representation, which has limited ability to represent complex scenes. To overcome these challenges, we introduce Lantent-SpecGS, an approach that utilizes a universal latent neural descriptor within each 3D Gaussian. This enables a more effective representation of 3D feature fields, including appearance and geometry. Moreover, two parallel CNNs are designed to decoder the splatting feature maps into diffuse color and specular color separately. A mask that depends on the viewpoint is learned to merge these two colors, resulting in the final rendered image. Experimental results demonstrate that our method obtains competitive performance in novel view synthesis and extends the ability of 3D-GS to handle intricate scenarios with specular reflections.

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Content] Media Interpretation

Relevance To Conference: One important contribution of the paper on 3D Gaussian splatting is the exploration of its potential applications in various fields such as computer graphics, computer vision, robotics, and autonomous navigation. The research emphasizes the optimization of 3D Gaussian splatting in terms of realism and physical modeling. Furthermore, the paper introduces new techniques to improve scene modeling, enable semantic tasks with 3D Gaussian splatting, and enhance a wide range of applications. These advancements have significant implications for the field of multimedia/multimodal processing and contribute to its progress.

Submission Number: 2351