Overcoming Domain Knowledge Forgetting in Continual Test-Time Adaptation via Siamese Networks

Zhihong Xu, Wenjun Shi, Dongchen Zhu, Xiaolin Zhang, Lei Wang, Jiamao Li

Abstract: Test-Time Adaptation (TTA) requires adapting a source-domain model to the target domain using online test data inputs. Existing methods that focus on adjusting normalization layers to swiftly adapt to a new domain often neglect the problem of domain knowledge forgetting, which hinders the model's generalization capability. To address this, we propose a novel Anti-forgetting Test-time Adaptation Network (ATAN) which consists of three Siamese networks---Forerunner, Bridge and Momentum. The bridge network transfers domain-specific knowledge from the forerunner network to the momentum network which effectively overcomes forgetting by integrating cross-domain knowledge. To further enhance the adaptability of the forerunner network, we propose reconstructing its loss function based on the voting information from the Siamese networks. To strengthen the learning of domain-invariant features, we introduce a weak augmentation consistency loss for the bridge network. Extensive experiments on corruption and natural shift datasets demonstrate the effectiveness and generalization of ATAN in long-term test-time domain adaptation scenarios.

Submission Number: 58