HMGS: Hybrid Model of Gaussian Splatting for Enhancing 3D Reconstruction with Reflections⋆

Wang Chengliang

Abstract:

The advent of 3D Gaussian Splatting (3D-GS) marks a sig
nificant breakthrough in the field of 3D reconstruction, leveraging GPU
rasterization technology to achieve real-time rendering with state-of-the
art quality. However, 3D-GS is limited by the capacity of low-order spher
ical harmonics to represent high-frequency reflective attributes, often re
sulting in the loss of critical information in scenes with highlights and
reflections. To address this limitation, we propose HMGS, a hybrid model
that enhances the original 3D-GS’s ability to capture reflective colors.
Our approach employs a neural network to learn color components from
both the camera viewing direction and the reflected light direction, which
are then jointly trained with the original 3D-GS model. Furthermore, we
introduce a smoothing loss for the viewing color component, effectively
decoupling the two color components. Our method significantly improves
the reconstruction performance of 3D-GS on datasets featuring metallic
sheen, light reflections, and shadows, while also enhancing reconstruction
quality on general datasets.