UniTTA: Unified Benchmark and Versatile Framework Towards Realistic Test-Time Adaptation

Chaoqun Du, Yulin Wang, Yizeng Han, Jiayi Guo, Jie Zhou, Gao Huang

Abstract:

Test-Time Adaptation (TTA) aims to adapt pre-trained models to the target domain
during testing. In reality, this adaptability can be influenced by multiple factors.
Researchers have identified various challenging scenarios and developed diverse
methods to address these challenges, such as dealing with continual domain shifts,
mixed domains, and temporally correlated or imbalanced class distributions. Despite these efforts, a unified and comprehensive benchmark has yet to be established.
To this end, we propose a Unified Test-Time Adaptation (UniTTA) benchmark,
which is comprehensive and widely applicable. Each scenario within the benchmark
is fully described by a Markov state transition matrix for sampling from the original
dataset. The UniTTA benchmark considers both domain and class as two independent dimensions of data and addresses various combinations of imbalance/balance
and i.i.d./non-i.i.d./continual conditions, covering a total of $(2\times3)^2 = 36$ scenarios.
It establishes a comprehensive evaluation benchmark for realistic TTA and provides
a guideline for practitioners to select the most suitable TTA method. Alongside this
benchmark, we propose a versatile UniTTA framework, which includes a Balanced
Domain Normalization (BDN) layer and a COrrelated Feature Adaptation (COFA)
method–designed to mitigate distribution gaps in domain and class, respectively.
Extensive experiments demonstrate that our UniTTA framework excels within the
UniTTA benchmark and achieves state-of-the-art performance on average. Our
code is available at https://github.com/LeapLabTHU/UniTTA.