MGSR: 2D/3D Mutual-boosted Gaussian Splatting for High-fidelity Surface Reconstruction under Various Light Conditions

Qingyuan Zhou, Yuehu Gong, Weidong Yang, Jiaze Li, Yeqi Luo, Baixin Xu, Shuhao Li, Ben Fei, Ying He

Abstract:

Novel view synthesis (NVS) and surface reconstruction (SR)
are essential tasks in 3D Gaussian Splatting (3DGS). Despite recent progress, these tasks are often addressed independently, with GS-based rendering methods struggling
under diverse light conditions and failing to produce accurate surfaces, while GS-based reconstruction methods
frequently compromise rendering quality. This raises a
central question: must rendering and reconstruction always involve a trade-off? To address this, we propose
MGSR, a 2D/3D Mutual-boosted Gaussian Splatting for
Surface Reconstruction that enhances both rendering quality and 3D reconstruction accuracy. MGSR introduces two
branches—one based on 2DGS and the other on 3DGS. The
2DGS branch excels in surface reconstruction, providing
precise geometry information to the 3DGS branch. Leveraging this geometry, the 3DGS branch employs a geometryguided illumination decomposition module that captures reflected and transmitted components, enabling realistic rendering under varied light conditions. Using the transmitted component as supervision, the 2DGS branch also
achieves high-fidelity surface reconstruction. Throughout
the optimization process, the 2DGS and 3DGS branches
undergo alternating optimization, providing mutual supervision. Prior to this, each branch completes an independent warm-up phase, with an early stopping strategy implemented to reduce computational costs. We evaluate MGSR
on a diverse set of synthetic and real-world datasets, at both
object and scene levels, demonstrating strong performance
in rendering and surface reconstruction. Code is available
at https://github.com/TsingyuanChou/MGSR.