Deep Learning with Data Privacy via Residual Perturbation

Wenqi Tao, Huaming Ling, Zuoqiang Shi, Bao Wang

Keywords: Data Privacy, Residual Perturbation, Deep Learning

Abstract: Protecting data privacy in deep learning (DL) is at its urgency. Several celebrated privacy notions have been established and used for privacy-preserving DL. However, many of the existing mechanisms achieve data privacy at the cost of significant utility degradation. In this paper, we propose a stochastic differential equation principled \emph{residual perturbation} for privacy-preserving DL, which injects Gaussian noise into each residual mapping of ResNets. Theoretically, we prove that residual perturbation guarantees differential privacy (DP) and reduces the generalization gap for DL. Empirically, we show that residual perturbation outperforms the state-of-the-art DP stochastic gradient descent (DPSGD) in both membership privacy protection and maintaining the DL models' utility. For instance, in the process of training ResNet8 for the IDC dataset classification, residual perturbation obtains an accuracy of 85.7\% and protects the perfect membership privacy; in contrast, DPSGD achieves an accuracy of 82.8\% and protects worse membership privacy.

One-sentence Summary: We propose a stochastic differential equation principled residual perturbation for privacy-preserving DL.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=YrPFMP-Vl