BaFTA: Backprop-Free Test-Time Adaptation for Zero-shot Vision Language Models

Xuefeng Hu, Ke Zhang, Min Sun, Albert Y. C. Chen, Cheng-Hao Kuo, Ram Nevatia

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: test time adaptation; training free; online clustering; vision language model;

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a simple but effective backpropagation-free test-time adaptation method to improve zero-shot classification ability of CLIP, and has outperformed the existing SOTA.

Abstract:

Large-scale pretrained vision-language models like CLIP have demonstrated remarkable zero-shot image classification capabilities across diverse domains. To enhance CLIP's performance while preserving the zero-shot paradigm, various test-time prompt tuning methods have been introduced to refine class embeddings through unsupervised learning objectives during inference. However, these methods often encounter challenges in selecting appropriate learning rates to prevent model instability with absence of validation data during test-time training.
In this study, we propose a novel backpropagation-free method for test-time adaptation in vision-language models. Instead of fine-tuning text prompts to refine class embeddings, our approach directly estimates class centroids using online clustering within a projected embedding space that aligns text and visual embeddings. We dynamically aggregate predictions from both estimated and original class embeddings, as well as from distinct augmented views, by assessing the reliability of each prediction using Rényi entropy.
Through extensive experimentation, we demonstrate that our approach consistently outperforms state-of-the-art test-time adaptation methods by a significant margin.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6294