Layer-wise Parameter Robustness for Continual Test-time Adaptation

Haoyu Xiong, Qiuxia Yang, Chengchao Wang, Tianze Zhong, Zhengpeng Zhao, Yuanyuan Pu

Abstract: Since inevitable distribution shifts are encountered during test time in practice, test-time adaptation (TTA) presents a promising solution by recalibrating the model online using only an unlabeled test data stream. However, TTA often suffers from issues such as catastrophic forgetting caused by continuously changing environments, as it relies on self-training. Contemporary solutions attempt to mitigate this by anchoring TTA to a static source model, such as stochastic parameter restoration or periodic parameter reset, which restrict model flexibility. Moreover, different layers may exhibit varying sensitivities to distribution shifts, sometimes even showing opposite shift trends, yet prior methods treat all layers homogeneously. Motivated by this, we propose a layer-wise parameter robustness method that autonomously identifies important parameters in different layers for selective weighting by measuring the sharpness of parameter surface. Further in-depth experiments on various benchmarks demonstrate the robustness and effectiveness of our proposed method.