Go to ICLR 2026 Conference homepageHeLutNet: Extremely Fast Privacy-preserving Inference in Milliseconds via LUT-based Machine Learning Models
Keywords: privacy preserving, fully homomorphic encryption
Abstract: Fully Homomorphic Encryption (FHE) holds immense promise for enabling privacy-preserving machine learning (PPML) inference, allowing computations on encrypted data without needing to expose sensitive information to untrusted third parties, such as cloud providers. Despite this potential, a major obstacle to FHE's widespread adoption is its prohibitively high computational overhead and slow inference speeds. For instance, the current state-of-the-art implementation requires 0.23 seconds for a single MNIST inference on one CPU core, a significant slowdown that renders FHE impractical for real-world applications. To overcome this, we introduce HeLutNet, an ultra-low-latency framework that accelerates FHE-based PPML inference by pioneering the use of lookup table (LUT)-based ML models. LUT-based models are uniquely suited for FHE, as they can represent complex non-linear functions without deep network architectures, enabling high accuracy on diverse datasets with shallow networks. Our approach converts a PyTorch-trained, LUT-based model into an efficient FHE program, implementing LUTs using the BGV scheme's SIMD capabilities and incorporating novel packing and LUT reordering strategies to minimize the number of homomorphic operations performed. Our evaluation across 20 vision, speech, health, and tabular datasets demonstrates consistent speedup and lower memory and communication overhead compared to state-of-the-art FHE-based PPML methods. Notably, we reduce inference time on various datasets to just 8.3 ms on a single-core CPU, achieving a 27.71x speedup over Orion (ASPLOS'25), thereby demonstrating the potential for a practical, real-time FHE-based PPML system.
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Submission Number: 23394
Loading