Efficient Gradient Clipping Methods in DP-SGD for Convolution Models
Keywords: Differential Privacy, SGD, Clipping, CNNs, FFT, DP-SGD, Computational Complexity
TL;DR: We provide computationally and memory efficient algorithms for gradient norm computation in CNNs that are used in DP-SGD.
Abstract: Differentially private stochastic gradient descent (DP-SGD) is a well-known method for training machine learning models with a specified level of privacy.
However, its basic implementation is generally bottlenecked by the computation of the gradient norm (gradient clipping) for each example in an input batch.
While various techniques have been developed to mitigate this issue,
there are only a handful of methods pertaining to convolution models, e.g., vision models.
In this work, we present three methods for performing gradient clipping that improve upon previous state-of-art methods. Two of these methods use in-place operations to reduce memory overhead, while the third one leverages a relationship between Fourier transforms and convolution layers.
To demonstrate the numerical efficiency of our methods, we also present several benchmark experiments that compare against other algorithms.
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2634
Loading