Stepwise High-Level Semantic Alignment for Test-Time Adaptation

Xizhong Yang, Ning Xu, Mofei Song

Keywords: stepwise semantic alignment, test-time adaptation, source distribution estimation

TL;DR: We propose a stepwise high-level semantic alignment framework with hierarchical feature aggregation and confidence-aware pseudo-label refinement for robust and stable test-time adaptation.

Abstract: Test-Time Adaptation (TTA) aims to adapt a source-trained model to a target domain without access to source data or target labels. Among existing approaches, Source Distribution Estimation (SDE) is valued for its ability to preserve source discriminability and ensure stable adaptation. However, most SDE-based methods rely on aligning low-level features statistics like batch normalization, often resulting in class confusion and unstable decision boundaries under large domain shifts. To address this, we propose **SHLSA**, a *Stepwise High-Level Semantic Alignment* framework that incorporates semantic priors to align features in the high-level space, preserving category structure and enabling more stable, semantically consistent adaptation. Specifically, SHLSA introduces the *pseudo-source domain* as a semantic bridge between the source and target domains, enabling a more stable and effective stepwise domain alignment (**SDA**) from reliable to ambiguous regions. To further enhance semantic feature quality, we design a hierarchical feature aggregation (**HFA**) module that integrates local and global representations via attention, improving local consistency and global convergence. Building on these enriched features, we introduce a confidence-aware complementary learning (**CACL**) strategy to refine EMA-updated pseudo-labels by suppressing noise and improving semantic reliability, thereby enhancing supervision for target domain samples. Extensive experiments on standard TTA benchmarks demonstrate the superior performance and generalizability of SHLSA.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 11315