DP-SGD Without Clipping: The Lipschitz Neural Network Way

Louis Béthune, Thomas Massena, Thibaut Boissin, Corentin Friedrich, Yannick Prudent, Franck Mamalet, Aurélien Bellet, Mathieu Serrurier, David Vigouroux

Keywords: privacy, lipschitz, dpsgd, gnp, neural, framework

TL;DR: Lipschitz neural networks can be trained under differential privacy guarantees without gradient clipping.

Abstract: State-of-the-art approaches for training Differentially Private (DP) Deep Neural Networks (DNN) faces difficulties to estimate tight bounds on the sensitivity of the network's layers, and instead rely on a process of per-sample gradient clipping. This clipping process not only biases the direction of gradients but also proves costly both in memory consumption and in computation cost. To provide sensitivity bounds and bypass the drawbacks of the clipping process, our theoretical analysis of Lipschitz constrained networks reveals an unexplored link between the Lipschitz constant with respect to their input and the one with respect to their parameters. By bounding the Lipschitz constant of each layer with respect to its parameters we guarantee DP training of these networks. This analysis not only allows computing the aforementioned sensitivities at scale but also provides leads on to how maximize the gradient to noise ratio for fixed privacy guarantees. To facilitate the application of Lipschitz networks and foster robust and certifiable learning under privacy guarantees, we provide a Python package that implements building blocks allowing the construction and private training of such networks.

Submission Number: 2847