3D Gaussian Splatting based Scene-independent Relocalization with Unidirectional and Bidirectional Feature Fusion

Junyi Wang, Yuze Wang, Wantong Duan, Meng Wang, Yue Qi

Keywords: Camera relocalization, 3D Gaussian Splatting, Feature Fusion

Abstract:

Visual localization is a critical component across various domains.
The recent emergence of novel scene representations, such as 3D Gaussian Splatting (3D GS), introduces new opportunities for advancing localization pipelines.
In this paper, we propose a novel 3D GS-based framework for RGB based, scene-independent camera relocalization, with three main contributions.
First, we design a two-stage pipeline with fully exploiting 3D GS.
The pipeline consists of an initial stage, which utilizes 2D-3D correspondences between image pixels and 3D Gaussians,
followed by pose refinement using the rendered image by 3D GS.
Second, we introduce a 3D GS based Relocalization Network, termed GS-RelocNet, to establish correspondences for initial camera pose estimation.
Additionally, we present a refinement network that further optimizes the camera pose.
Third, we propose a unidirectional 2D-3D feature fusion module and a bidirectional image feature fusion module, integrated into GS-RelocNet and the refinement network, respectively, to enhance feature sharing across the two stages.
Experimental results on public 7 Scenes, Cambridge Landmarks, TUM RGB-D and Bonn demonstrate state-of-the-art performance.
Furthermore, the beneficial effects of the two feature fusion modules and pose refinement are also highlighted.
In summary, we believe that the proposed framework can be a novel universal localization pipeline for further research.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 9714