Reducing Task Discrepancy of text encoders for Zero-Shot Composed Image Retrieval

Jaeseok Byun, Seokhyeon Jeong, Wonjae Kim, Sanghyuk Chun, Taesup Moon

Keywords: composed image retrieval; language only supervision; task discrepancy

Abstract:

Composed Image Retrieval (CIR) aims to retrieve a target image based on a reference image and conditioning text, enabling controllable image searches.
Due to the expensive dataset construction cost for CIR triplets, a zero-shot (ZS) CIR setting has been actively studied to eliminate the need for human-collected triplet training datasets of the target domain.
The mainstream methods of ZS-CIR research typically employ a projection module that projects a CLIP image embedding to the CLIP text token embedding space while all encoders are fixed.
Using such a projected embedding, those methods then generate an image-text composed feature, which is used as a query for retrieval.
However, we point out that using fixed CLIP encoders for ZS-CIR has an inherent limitation since there exists a significant task discrepancy between the original pre-training task of the encoders (text $\leftrightarrow$ image) and the target CIR task (image + text $\leftrightarrow$ image). To reduce such a discrepancy, a naive solution would be to train both image and text encoders with CIR triplets in a supervised manner.
Instead, we introduce the Reducing Task Discrepancy of text encoders for Zero-Shot Composed Image Retrieval (RTD), an efficient post-precessing approach designed to enhance the capability of text encoders for ZS-CIR. Namely, we devise a novel target-anchored text contrastive learning, which solely updates the text encoder using cheap text triplets, consisting of reference and target texts instead of images.
We also introduce two enhancements to this approach: a refined batch sampling strategy and a sophisticated concatenation scheme.
Integrating RTD into existing projection-based ZS-CIR methods significantly improves performance across various datasets and backbones, achieving competitive or superior results compared to other resource-intensive state-of-the-art CIR methods beyond projection-based approaches.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5735