ICE: Image-Caption Encoding for Improved Out-Of-Distribution Generalization In Vision-Language Models

Eric Yang Yu, Christopher Liao, Sathvik Ravi, Theodoros Tsiligkaridis, Brian Kulis

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: zero shot classification, image-caption encoding

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a novel zero-shot classification method that easily combines with other state-of-the-art methods for increased performance.

Abstract:

Recent advances in vision-language models have combined contrastive approaches with generative methods to achieve state-of-the-art (SOTA) on downstream inference tasks like zero-shot image classification.
However, one persistent issue of these models for image classification is their out-of-distribution (OOD) generalization capabilities.
We first show that when an OOD datapoint is misclassified, the correct class can be typically found in the Top-$K$ predicted classes.
In order to steer the model prediction toward the correct class within the top predicted classes, we propose the Image-Caption Encoding (ICE) method, a straightforward approach that directly enforces consistency between the image-conditioned and caption-conditioned predictions at evaluation time only.
Intuitively, we take advantage of unique properties of the generated captions to guide our local search for the correct class label within the Top-$K$ predicted classes.
We show that our method can be easily combined with other SOTA methods to enhance Top-1 OOD accuracies by 0.5% on average and up to 3% on challenging datasets.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3752