Enabling Secure Cross-Modal Search Over Encrypted Data via Federated Learning

Xiaoming Wang, Jianwei Li, Zhiquan Liu, Quan Tang, Xixian Wang

Abstract: Cross-modal search with deep learning shows an attractive potential on heterogeneous data sets due to its accuracy and effectiveness. Usually, it has to aggregate and train large amounts of data to make precise predictions, which is feasible in the single-user environment and plaintext areas. However, the challenge lies in maintaining search accuracy within a multiuser environment while simultaneously safeguarding user data privacy. In this article, we address these challenges by centering on the development of secure cross-modal search techniques that are supported by federated learning. To our knowledge, this marks the first endeavor to execute cross-modal encrypted data search through the auspices of federated learning. Our approach specifically employs a secure and reliable federated learning technique to extract key features from diverse heterogeneous data, ensuring precise model training in a distributed data environment. Consequently, while the data is encrypted, it achieves the protection of data privacy and simultaneously enhances the efficiency and accuracy of cross-modal search. To further enhance retrieval efficiency, we propose a tag classification algorithm that employs homomorphic encryption and locality-sensitive hashing. Furthermore, we design a secure method for calculating Euclidean distance that utilizes the k-nearest neighbor algorithm. This method efficiently identifies the result nearest to the query data, thereby enhancing the accuracy of the search results. Both theoretical analysis and experimental evaluation demonstrate that our proposed scheme not only protects user data privacy but also has high accuracy and efficiency.