Efficient Open Set Single Image Test Time Adaptation of Vision Language Models
Abstract: Adapting models to dynamic, real-world environments characterized by shifting data distributions and unseen test scenarios is a critical challenge in deep learning. In this paper, we consider a realistic and challenging Test-Time Adaptation setting, where a model must continuously adapt to test samples that arrive sequentially, one at a time, while distinguishing between known and unknown classes. Current Test-Time Adaptation methods operate under closed-set assumptions or batch processing, differing from the real-world open-set scenarios. We address this limitation by establishing a comprehensive benchmark for Open-set Single-image Test-Time Adaptation using Vision-Language Models. Furthermore, we propose ROSITA, a novel framework that leverages dynamically updated feature banks to identify reliable test samples and employs a contrastive learning objective to improve the separation between known and unknown classes. Our approach effectively adapts models to domain shifts for known classes while rejecting unfamiliar samples. Extensive experiments across diverse real-world benchmarks demonstrate that ROSITA sets a new state-of-the-art in open-set TTA, achieving both strong performance and computational efficiency for real-time deployment. The code is released at https://github.com/manogna-s/ROSITA.git.
Submission Length: Regular submission (no more than 12 pages of main content)
Code: https://github.com/manogna-s/ROSITA.git
Assigned Action Editor: ~Steffen_Schneider1
Submission Number: 3976
Loading