Test-Real-Time Adaptation against Sparse Knowledge Bottleneck

Guowei Wang, Yan Huang, Fan Lyu, Changxing Ding

Keywords: Test-time adaptation, online adaptation

TL;DR: This paper proposes a novel agreement- and uncertainty-guided reweighting to address sparse-knowledge bottleneck for a new realistic test-real-time adaptation paradigm.

Abstract: Test-time adaptation (TTA) typically involves adaptation delays due to self-training, which conflict with real-time deployment where inference cannot pause for adaptation. We introduce Test-Real-Time Adaptation (TRTA), which requires uninterrupted prediction while adaptation runs in the background, leaving few update opportunities. In TTA, later reliable signals enable error correction and steady knowledge accumulation, whereas in TRTA, such signals are rare, so knowledge growth stalls. We term this the sparse-knowledge bottleneck, where limited updates hinder error correction and increase the risk in self-training. To solve this challenge, we propose a novel method, dubbed {A}greement- and {U}ncertainty-{G}uided {R}eweighting ({AUGR}). AUGR fuses two complementary evidence sources: \textit{(i)} inter-model agreement, defined as the concordance of predicted class rankings between the base and the reference models on each sample, revealing common knowledge with consensus predictions; and \textit{(ii)} inner-model uncertainty, representing the reliability of such knowledge, which balance the agreement evidence by discounting low-confidence cases. By integrating both sources of evidence, AUGR emphasizes the learning of consistent, reliable samples and suppresses conflicting or uncertain ones, thereby promoting robust knowledge accumulation. Extensive experiments on ImageNet-C/R/K demonstrate the effectiveness of AUGR combating sparse-knowledge bottleneck in TRTA. Code will be released.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 10799