Keyword Spotting in the Homomorphic Encrypted Domain Using Deep Complex-Valued CNN

Zhiwei Cai

Abstract: In this paper, we propose a non-interactive scheme to achieve endto-end keyword spotting in the homomorphic encrypted domain using deep learning techniques. We carefully designed a complexvalued convolutional neural network (CNN) structure for the encrypted domain keyword spotting to take full advantage of the limited multiplicative depth. At the same depth, the proposed complexvalued CNN can learn more speech representations than the realvalued CNN, thus achieving higher accuracy in keyword spotting. The complex activation function of the complex-valued CNN is nonarithmetic and cannot be supported by homomorphic encryption. To implement the complex activation function in the encrypted domain without interaction, we design methods to approximate complex activation functions with low-degree polynomials while preserving the keyword spotting performance. Our scheme supports single-instruction multiple-data (SIMD), which reduces the total size of ciphertexts and improves computational efficiency. We conducted extensive experiments to investigate our performance with various metrics, such as accuracy, robustness, and F1-score. The experimental results show that our approach significantly outperforms the state-of-the-art solutions on every metric.