Test-time Adaptation for Cross-modal Retrieval with Query Shift

Haobin Li, Peng Hu, Qianjun Zhang, Xi Peng, XitingLiu, Mouxing Yang

Keywords: Test-time adaptation, Cross-modal retrieval, Query shift

TL;DR: We propose a novel test-time adaptation method for achieving robust cross-modal retrieval against query shift.

Abstract:

The success of most existing cross-modal retrieval methods heavily relies on the assumption that the given queries follow the same distribution of the source domain.
However, such an assumption is easily violated in real-world scenarios due to the complexity and diversity of queries, thus leading to the query shift problem.
Specifically, query shift refers to the online query stream originating from the domain that follows a different distribution with the source one.
In this paper, we observe that query shift would not only diminish the uniformity (namely, within-modality scatter) of the query modality but also amplify the gap between query and gallery modalities.
Based on the observations, we propose a novel method dubbed Test-time adaptation for Cross-modal Retrieval (TCR).
In brief, TCR employs a novel module to refine the query predictions (namely, retrieval results of the query) and a joint objective to prevent query shift from disturbing the common space, thus achieving online adaptation for the cross-modal retrieval models with query shift.
Expensive experiments demonstrate the effectiveness of the proposed TCR against query shift.
Code is available at https://github.com/XLearning-SCU/2025-ICLR-TCR.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10708