Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models

Konstantinos M. Dafnis, Dimitris N. Metaxas

Keywords: Vision-Language Models, Foundation Model, CLIP, Test-Time Adaptation, Generalization, Out-of-Distribution Robustness

TL;DR: We propose Test-Time Spectrum-Aware Latent Steering (STS), a lightweight adaptation that learns a few per-sample steering parameters in latent space to align visual and text embeddings and boost zero-shot robustness.

Abstract: Vision–language models (VLMs) excel at zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strategies have recently emerged as powerful techniques for adapting VLMs to a single unlabeled image. However, existing adaptation strategies, such as test-time prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce \textbf{S}pectrum-Aware \textbf{T}est-Time \textbf{S}teering (\textbf{STS}), a \textit{lightweight adaptation framework} that extracts a spectral subspace from the textual embeddings to define principal semantic directions, and learns to steer latent representations in a spectrum-aware manner by adapting a small number of per-sample shift parameters to minimize entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8× faster with a 12× smaller memory footprint than conventional test-time prompt tuning. The code is available at \url{https://github.com/kdafnis/STS}.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 24791