Plug-In Open-Set Cross-Modal Hashing

Bowen Wang, Lei Zhu, Fengling Li, Hui Cui, Jingjing Li

Abstract: Unsupervised Cross-Modal Hashing (UCMH) models the intrinsic semantic correlations across different modalities to generate binary hash codes, facilitating efficient cross-modal retrieval. This technology offers notable advantages, such as independence from labeled data and superior generalization capabilities compared to supervised methods. However, most UCMH methods are designed for closed-set retrieval scenarios and have difficulty generalizing to open multi-modal data, which is common in real-world retrieval settings. This limitation hampers their performance in open retrieval tasks, particularly when these tasks involve novel categories. To address the above issue, we propose an Open-set Cross-Modal Hashing (OCMH) method, which enhances the generalization capability of trained UCMH models in an efficient plug-in manner for open cross-modal retrieval. Our method enables the model to learn from novel categories in open-set scenarios by increasing the pre-defined hash code length, while simultaneously preventing the catastrophic forgetting of trained knowledge from the closed-set domain using basic hash codes. Additionally, we introduce a historical-category detection module and an asymmetric optimization strategy to support the joint learning of basic and increased hash codes by replaying detected samples related to historical categories. By plugging our proposed method into several representative UCMH methods on three widely used datasets, experimental results show that the enhanced UCMH methods achieve superior retrieval performance in both open-set and closed-set scenarios.

External IDs:dblp:journals/tmm/WangZLCL25