FedCAFE: Federated Cross-Modal Hashing with Adaptive Feature Enhancement

Ting Fu, Yu-Wei Zhan, Chong-Yu Zhang, Xin Luo, Zhen-Duo Chen, Yongxin Wang, Xun Yang, Xin-Shun Xu

Abstract: Deep Cross-Modal Hashing (CMH) has become one of the most popular solutions for cross-modal retrieval. Existing methods need to first collect data and then be trained with these accumulated data. However, in real world, data may be generated and possessed by different owners. Considering the concerns about privacy, data may not be shared or transmitted, leading to the failure of sufficient training of CMH. To solve the problem, we propose a new framework called Federated Cross-modal Hashing with Adaptive Feature Enhancement (FedCAFE). FedCAFE is a federated method which could use distributed data to train existing CMH methods under the privacy protection. To overcome the data heterogeneity challenge of distributed data and improve the generalization ability of global model, FedCAFE is endowed with a novel adaptive feature enhancement module and a new weighted aggregation strategy. Besides, it could fully utilize the rich global information carried in the global model to constrain the model during the local training process. We have conducted extensive experiments on four widely-used datasets in CMH domain with both IID and non-IID settings. The reported results demonstrate that the proposed FedCAFE achieves better performance than several state-of-the-art baselines. As the topic that training deep CMH in federated scenario is in its infancy, we plan to release the code and data to boost the development of the field. However, considering restriction of anonymous submission and size limitation, we could only upload the source code of FedCAFE as supplementary materials for peer review at the present stage.

Primary Subject Area: [Engagement] Multimedia Search and Recommendation

Secondary Subject Area: [Systems] Data Systems Management and Indexing, [Experience] Multimedia Applications

Relevance To Conference: The continuous advancement of deep learning technology has greatly propelled the development of cross-modal hashing retrieval. However, the escalating concerns regarding data security have increasingly become a significant obstacle to further progress in this field. To address this challenge, this paper delves into the study of federated learning technology and innovatively proposes a federated cross-modal hashing retrieval method based on adaptive feature enhancement. By effectively integrating federated learning with deep cross-modal hashing retrieval tasks, this method not only enhances retrieval accuracy but also provides a practical solution to privacy and security concerns in real-world applications.

Submission Number: 3367