Data Efficient Subset Training with Differential Privacy

ICLR 2025 Workshop BuildingTrust Submission135 Authors

Track: Tiny Paper Track (between 2 and 4 pages)

Keywords: Data Efficient Training, Differential Privacy

TL;DR: Private machine learning hinders training convergence due to noisy training process and is compute heavy due to per sample gradient computation. We explore data efficient subset training approach to make private training efficient.

Abstract:

Private machine learning introduces a trade-off between the privacy budget and
training performance. Training convergence is substantially slower and extensive
hyper parameter tuning is required. Consequently, efficient methods to conduct
private training of models is thoroughly investigated in the literature. To this end,
we investigate the strength of the data efficient model training methods in the
private training setting. We adapt GLISTER (Killamsetty et al., 2021b) to the
private setting and extensively assess its performance. We empirically find that
practical choices of privacy budgets are too restrictive for data efficient training in
the private setting. Our code can be found here.

Submission Number: 135