On Theoretical Limits of Learning with Label Differential Privacy

Puning Zhao, Huiwen Wu, Jiafei Wu, Zhe Liu

Keywords: label differential privacy

Abstract: Label differential privacy (DP) is designed for learning problems with private labels and public features. Although various methods have been proposed for learning under label DP, the theoretical limits remain unknown. The main challenge is to take infimum over all possible learners with arbitrary model complexity. In this paper, we investigate the fundamental limits of learning with label DP under both central and local models. To overcome the challenge above, we derive new lower bounds on testing errors that are adaptive to the model complexity. Our analyses indicate that $\epsilon$-local label DP only enlarges the sample complexity with respect to $\epsilon$, without affecting the convergence rate over the sample size $N$, except the case with heavy-tailed label. Under the central model, the performance loss due to the privacy mechanism is further weakened, such that the additional sample complexity becomes negligible. Overall, our analysis validates the promise of learning under the label DP from a theoretical perspective and shows that the learning performance can be significantly improved by weakening the DP definition to only labels.

Primary Area: Learning theory

Submission Number: 10110