Compensating for Nonlinear Reduction with Linear Computations in Private Inference

Fabing Li, Yuanhao Zhai, Mingyu Gao

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Privacy Preserving Machine Learning; Network Architecture Search;Cryptography;

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Increasingly serious data privacy concerns and strict regulations have recently posed significant challenges to machine learning, a field that hinges on high-performance processing of massive user data.
Consequently, privacy-preserving machine learning (PPML) has emerged to securely execute machine learning tasks without violating privacy.
Unfortunately, the computational cost to securely execute nonlinear computations in PPML models remains significant, calling for new neural architecture designs with fewer nonlinear operations.
We propose Seesaw, a novel neural architecture search method tailored for PPML.
Seesaw exploits a previously unexplored opportunity to leverage more linear computations and nonlinear result reuse, in order to compensate for the accuracy loss due to nonlinear reduction.
It also incorporates specifically designed pruning and search strategies to efficiently handle the much larger design space including both nonlinear and linear operators.
Compared to the previous state-of-the-art PPML for image classification on ImageNet, Seesaw achieves $1.68\times$ less latency at 71\% iso-accuracy, or 4.59\% higher accuracy at iso-latency of 1000K ReLU operations.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5216