Dynamic Model Fusion for Multi-Source Test-Time Adaptation

Yuan Xue, Qinting Jiang, Yuan Meng, Xingxuan Zhang, Chen Tang, Jingyan Jiang, Zhi Wang

Abstract: Deep Neural Networks suffer significant performance degradation when faced with distribution shifts between training and test data. Test-time adaptation (TTA) has emerged as a practical solution that enables models to adapt to the shifted test distribution. Currently, most existing TTA methods are designed around a single model, which incorporate limited information from a singular data distribution. In practice, pre-trained models derived from diverse source domains are readily accessible, each capturing a distinct data distribution and containing complementary information. To exploit this diversity, we propose Model Fusion-based multi-source Test-Time Adaptation (MFTTA), which constructs a target model by fusing the parameters of multiple source models. Drawing inspiration from deep model fusion, we introduce a fine-grained fusion mechanism governed by an off-policy reinforcement learning agent, which dynamically assigns fusion weights based on the current data distribution. Furthermore, we design a correlation-aware model update strategy that prioritizes the source model most relevant to the incoming test data. Extensive experiments on standard out-of-distribution benchmarks demonstrate that our method effectively integrates knowledge from multiple source models, adapts robustly to dynamic distribution shifts, and alleviates the problem of forgetting in long-term adaptation.

External IDs:dblp:conf/ecai/XueJMZTJ025