ULD-Net: Enabling Ultra-Low-Degree Fully Polynomial Networks for Homomorphically Encrypted Inference

ICLR 2026 Conference Submission22477 Authors

Keywords: Privacy-Preserving Machine Learning, efficient private inference, machine learning as a service, homomorphic encryption, Fully Polynomial Networks, Ultra-Low-Degree operators

Abstract:

Fully polynomial neural networks—models whose computations comprise only additions and multiplications—are attractive for privacy-preserving inference under homomorphic encryption (HE). Yet most prior systems obtain such models by *post-hoc* replacement of nonlinearities with high-degree or cascaded polynomials, which inflates HE cost and makes training numerically fragile and hard to scale.

We introduce **ULD-Net**, a pretraining methodology that enables *ultra-low-degree* (multiplicative depth ≤ 3 for each operator) fully polynomial networks to be trained from scratch at ImageNet and transformer scale while maintaining high accuracy. The key is a polynomial-only normalization, **PolyNorm**, coupled with a principled choice of normalization axis that keeps activations in a well-conditioned range across deep stacks of polynomial layers. Together with a special set of polynomial-aware operator replacements, such as polynomial activation functions and linear attention, ULD-Net delivers stable optimization without resorting to high-degree approximations.

Experimental results demonstrate that ULD-Net outperforms several state-of-the-art open-source fully and partially polynomial approaches across both CNNs and ViTs on diverse datasets, in terms of both accuracy and HE inference latency. Specifically, ULD-Net achieves +0.39% accuracy and a 2.76× speedup compared to the best fully polynomial baseline; up to +3.33% accuracy and a 3.17× speedup compared to the best partial polynomial baseline; and up to +0.88% accuracy and a 20.5× reduction in non-polynomial operator cost compared to the best HE transformer baseline. Applying ULD-Net to the VanillaNet family yields up to 76.40% top-1 accuracy on ImageNet with substantially reduced HE latency.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 22477