Neural Manifold Anchoring for Drift-Free Visual Relocalization in Dynamic Scenes

Tony Wang

Abstract: Visual relocalization methods based on scene coordinate regression have achieved impressive accuracy in static benchmarks, yet their transition to real-world robotic navigation reveals a persistent fragility: per-frame predictions, while individually precise, produce spatially incoherent trajectories that violate the kinematic continuity expected by downstream planning modules. This discrepancy between frame-level accuracy and trajectory-level smoothness constitutes a fundamental deployment barrier. We introduce a relocalization framework grounded in manifold-constrained optimization to address this gap. Central to our approach is a pretrained neural implicit surface representation of the environment, which provides a continuous, differentiable geometric scaffold. Rather than treating the network outputs as final position estimates, we reinterpret them as noisy observations that must be projected onto the learned scene manifold through a projection residual minimization scheme. This spatial anchoring is complemented by a lightweight temporal propagator that distills the iterative optimization process into a closed-form kinematic module, eliminating the computational overhead of online implicit field queries. A gated state estimation layer then reconciles the geometrically anchored spatial hypotheses with the temporally propagated kinematic states in a fully differentiable manner, enabling end-to-end training of the entire pipeline. Experiments on multiple visual localization benchmarks demonstrate that our method eliminates trajectory-level jitter without sacrificing point-wise accuracy, achieving state-of-the-art performance under both static and dynamic scene conditions.