A Descriptor-Based Multi-Cluster Memory for Test-Time Adaptation

Xingyi Zheng, Yu-Wen Tseng, Ya-Chen Wu, Jhih-Ciang Wu, I-Bin Liao, Yung-Hui Li, Hong-Han Shuai, Wen-Huang Cheng

Keywords: Test-time Adaptation, Memory Bank, Efficiency, Statistical Descriptor

Abstract: Test-time adaptation (TTA) aims to preserve model robustness under distribution shifts without access to source data. However, existing memory designs, often based on single clusters or naive sample storage, struggle to capture the diversity of target distributions and adapt efficiently over time. We introduce Multi-Cluster Memory (MCM), a novel memory management framework that organizes samples into multiple clusters using lightweight statistical descriptors such as sample means and variances. The inter-cluster distance naturally expands the coverage of the sample distribution, supports on-demand cluster creation for novel patterns, and maintains bounded capacity through an Adjacent Cluster Consolidation (ACC) mechanism that merges neighbor clusters in descriptor space. To further strengthen adaptation, we propose Relevance-guided Sample Retrieval (RSR), which selects the most target domain-relevant clusters for learning and integrates them into a Mean-Teacher self-supervised paradigm. Extensive experiments across CIFAR-10/100-C, ImageNet-C, and DomainNet demonstrate that MCM consistently outperforms prior methods under Practical TTA (PTTA) and achieves sustained robustness in recurring TTA. By delivering a memory structure that is more representative, scalable, and adaptive, MCM establishes multi-cluster memory as a practical and effective foundation for real-world test-time adaptation.

Primary Area: learning on time series and dynamical systems

Submission Number: 2876