Isolated Aggregators: Towards Forgetting-Free Continual Visual Place Recognition with Fast Adaptation

Dongshuo Zhang, Ze Yang, Edward Tan Beng Wai, Siew Kei Lam, Meiqing Wu

Keywords: Visual Place Recognition, Continual Learning

Abstract: Visual Place Recognition (VPR), the task of identifying revisited places by a query image, suffers significant degradation in long-term deployment due to non-stationary distribution shifts. Existing methods mainly rely on regularization- and/or replay-based continual learning strategies to address this challenge. However, regularization remains vulnerable to catastrophic forgetting under strong domain shifts, while replay introduces additional storage and latency costs and raises privacy concerns, making online adaptation impractical. To this end, we propose Isolated Aggregators, a new paradigm where each new environment is assigned an independent aggregator following a shared, frozen backbone. By design, parameters for the backbone and all previously learned aggregators are frozen, providing a structural guarantee against catastrophic forgetting. Meanwhile, finetuning only the new, lightweight aggregator for the current domain enables fast, privacy-preserving online adaptation to new environments without replay. We further maintain domain descriptors that allow the model to automatically select the appropriate aggregator during inference, ensuring robust continual VPR across diverse environments. Extensive experiments show that our method achieves both zero forgetting and fast adaptation, improving Recall@1 by +9.9\% (city-like to nature) and +3.5\% (nature to indoor) and within just around 30 and 90 seconds of single-epoch and single-pass training on an NVIDIA RTX 4090.

Primary Area: applications to robotics, autonomy, planning

Submission Number: 5706