GOTTA be diverse

Shyma Yaser Alhuwaider, Yasmeen Youssef Alsaedy, Silvio Giancola, Bernard Ghanem

Keywords: test time adaptation, domain adaptation, computer vision

Abstract: Test-Time Adaptation (TTA) enables models to adjust to distribution shifts using only the incoming test stream. While existing methods perform well under covariate shifts, their performance drops when label distributions also change, a common scenario in real-world streams. Some approaches attempt to mitigate this by introducing memory modules into their methods, typically to enforce class balance. However, because these memories are evaluated only in conjunction with specific algorithms, their independent role and effectiveness remain unclear. In this work, we systematically study memory in TTA by decoupling it from the adaptation algorithm. Through a unified evaluation, we identify the design choices that make memory effective under different stream settings. Building on these insights, we propose Guided Observational Test-Time Adaptation (GOTTA), a category of diversity-aware memories that combine class balance with intra-class diversity. Our results show that such memories provide reliable, compact, and efficient support for adaptation in dynamic test streams, highlighting diversity-aware memory as an important principle for robust TTA.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 25178