Polynomial Adaptation of Large-Scale CNNs for Homomorphic Encryption-Based Secure Inference

Moran Baruch, Nir Drucker, Gilad Ezov, Yoav Goldberg, Eyal Kushnir, Jenny Lerner, Omri Soceanu, Itamar Zimerman

Abstract: Enabling secure inference of large-scale CNNs using Homomorphic Encryption (HE) requires a preliminary step for adapting unencrypted pre-trained models to only use polynomial operations. Prior art advocates for high-degree polynomials for accurate approximations, which comes at the price of extensive computations. We demonstrate that low-degree polynomials can be sufficient for accurate approximation even for large-scale DNNs. For that, we introduce a dedicated fine-tuning process on unencrypted data that reduces the input range to the activation functions. The resulting models have competitive accuracy of up to 3.5% degradation from the original non-polynomial model, which outperforms prior art on tasks such as ImageNet classification over ResNet and ConvNeXt. Upon adaptation, these models can process HE-encrypted samples and are ready for secure inference. Based on these, we provide optimization insights for activation functions and skip connections, enhancing HE evaluation efficiency.