Decentralizing Test-time Adaptation under Heterogeneous Data Streams

Zixian Su, Jingwei Guo, Xi Yang, Qiufeng Wang, Kaizhu Huang

Keywords: Test-time Adaptation, Transfer Learning, Data Heterogeneity

Abstract:

While Test-Time Adaptation (TTA) has shown promise in addressing distribution shifts between training and testing data, its effectiveness diminishes with heterogenous data streams due to uniform target estimation. As previous attempts merely stabilize model fine-tuning over time to handle continually changing environments, they fundamentally assume a homogeneous target domain at any moment, leaving the intrinsic real-world data heterogeneity unresolved. This paper delves into TTA under heterogeneous data streams, moving beyond current model-centric limitations. By revisiting TTA from a data-centric perspective, we discover that decomposing samples into Fourier space facilitates an accurate data separation across different frequency levels. Drawing from this insight, we propose a novel Frequency-based Decentralized Adaptation framework, which transitions data from globally heterogeneous to locally homogeneous in Fourier space and employs decentralized adaptation to manage diverse distribution shifts.
Particularly, multiple local models are allowed to independently adjust to their specific data segments while periodically exchanging knowledge to form a cohesive global model. As such, not only can data diversity be captured, but also the overall model generalization can be enhanced across multiple distribution shifts. Importantly, we devise a novel Fourier-based augmentation strategy to assist in decentralizing adaptation, which selectively augments samples for each type of distribution shift and further enhances model robustness in complex real-world environments. Extensive experiments across various settings (corrupted, natural, and medical) demonstrate the superiority of our proposed framework over the state-of-the-arts.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3168