Visual Sparse Steering (VS2): Unsupervised Adaptation for Image Classification via Sparsity-Guided Steering Vectors

ICLR 2026 Conference Submission21095 Authors

Keywords: Sparse Autoencoders, Steering Vectors, Unsupervised Adaptation, Image Classification

Abstract: Steering vision foundation models at test time, without retraining or access to large labeled datasets, is a desirable yet challenging goal, particularly in dynamic or resource-constrained settings. We present Visual Sparse Steering (VS2), a lightweight, label-free test-time method that constructs a steering vector from sparse features extracted by a Sparse Autoencoder (SAE) trained on the model’s internal activations. On CIFAR-100, CUB-200, and Tiny-ImageNet, VS2 improves the top-1 accuracy of the CLIP zero-shot baseline by 4.12\%, 1.08\%, and 1.84\%, respectively. Since not all features learned by the SAE are equally important for classification, we introduce VS2++, a retrieval-augmented variant that selectively amplifies relevant sparse features using pseudo-labeled neighbors retrieved from an external unlabeled corpus at inference time. With oracle positive and negative sets (upper bound), VS2++ achieves absolute top-1 gains over the CLIP zero-shot baseline of up to 21.44\% on CIFAR-100, 7.08\% on CUB-200, and 20.47\% on Tiny-ImageNet, highlighting the potential of steering vectors when relevant feature selection is accurate. VS2 and VS2++ also improve per-class accuracy by up to 25\% and 38\%, respectively, indicating that sparse steering disproportionately benefits visually or semantically similar classes. Finally, VS2 includes a built-in reliability diagnostic based on SAE reconstruction loss, which is absent in common steering vectors, signaling when steering may underperform and safely triggering a fallback to the baseline.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 21095