xMLP: Revolutionizing Private Inference with Exclusive Square Activation
Primary Area: societal considerations including fairness, safety, privacy
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Privacy Preserving Machine Learning, Private Inference, Multi-Party Computation, Deep Learning
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: This paper proposes, xMLP, a DNN architecture using only multiplications and additions outperforming ReLU-based DNNs, and achieving 7x faster Private Inference speed on CPU (700x on GPU) compared to state-of-the-art PI models with similar accuracy.
Abstract: Private Inference (PI) enables deep neural networks (DNNs) to work on private data without leaking sensitive information by exploiting cryptographic primitives such as multi-party computation (MPC) and homomorphic encryption (HE).
However, the use of non-linear activations such as ReLU in DNNs can lead to impractically high PI latency in existing PI systems, as ReLU requires the use of costly MPC computations, such as Garbled Circuits.
Since square activations can be processed by Beaver's triples hundreds of times faster compared to ReLU, they are more friendly to PI tasks, but using them leads to a notable drop in model accuracy.
This paper starts by exploring the reason for such an accuracy drop after using square activations, and concludes that this is due to an ``information compounding’’ effect. Leveraging this insight, we propose xMLP, a novel DNN architecture that uses square activations exclusively while maintaining parity in both accuracy and efficiency with ReLU-based DNNs.
Our experiments on CIFAR-100 and ImageNet show that xMLP models consistently achieve better performance than ResNet models with fewer activation layers and parameters while maintaining consistent performance with its ReLU-based variants.
Remarkably, when compared to state-of-the-art PI Models, xMLP demonstrates superior performance, achieving a 0.58\% increase in accuracy with 7$\times$ faster PI speed. Moreover, it delivers a significant accuracy improvement of 4.96\% while maintaining the same PI latency.
When offloading PI to the GPU, xMLP is up to 700$\times$ faster than the previous state-of-the-art PI model with comparable accuracy.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 1384
Loading