Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Sensor Degradation

TMLR Paper8952 Authors

Abstract:

Pre-trained vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated a remarkable generalizability, enabling a wide range of applications, including zero-shot classification.
However, vision-language models still struggle to handle distribution shifts, where input samples have large gaps from training ones.
We found that CLIP is especially vulnerable to sensor degradation, a type of realistic distribution shift caused by sensor conditions such as weather, light, or noise.
Collecting a new dataset from a test distribution for fine-tuning is highly costly since sensor degradation occurs unexpectedly and has a wide variety of types.
Thus, we investigate test-time adaptation (TTA) of zero-shot classification, which enables on-the-fly adaptation to the test distribution with unlabeled test data.
Existing TTA methods for CLIP mainly focus on modifying image and text embeddings or predictions to address distribution shifts.
Although these methods can adapt to domain shifts, such as out-of-distribution or different renditions in input images, they fail to adapt to distribution shifts beyond domain shifts, e.g., sensor degradation.
We found that uniformity of image embeddings, which is related to the amount of information, is a key factor that differentiates domain shifts and other distribution shifts.
To enable adaptation on distribution shifts including sensor degradation, we propose a novel method called uniformity-aware information-balanced TTA (UnInfo).
To address distribution shifts, we introduce uniformity-aware confidence maximization, information-aware loss balancing, and knowledge distillation from the exponential moving average (EMA) teacher.
Through experiments, we demonstrate that our UnInfo improves accuracy under sensor degradation by retaining information in terms of uniformity.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Massimiliano_Mancini1

Submission Number: 8952