LMO-DP: Accurately Fine-Tuning Language Models with Stronger Differential Privacy

Qin Yang, Meisam Mohammady, Han Wang, Ali Payani, Ashish Kundu, Kai Shu, Yan Yan, Yuan Hong

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Differential Privacy, Natural Language Processing, Fine-tuning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Differentially Private Stochastic Gradient Descent (DP-SGD) and its variants have been proposed to ensure rigorous privacy for fine-tuning large-scale pre-trained language models. State-of-the-art (SOTA) DP-SGD methods rely heavily on the Gaussian mechanism since its key component – moment accountant (MA) leverages the properties of Gaussian noise to accumulate the overall privacy budget via tight DP composition. However, the privacy constraints imposed in DP-SGD, solely on the Gaussian noise, may still overly perturb the gradients and degrade the fine-tuning accuracy, especially in stronger privacy regimes (e.g., the total privacy budget $\epsilon < 3$). To address such limitations, we propose a novel Language Model-based Optimal Differential Privacy (LMO-DP) framework, which takes the first step to enable the tight composition of a sub-optimal DP mechanism (non-Gaussian) for accurately fine-tuning language models, even in stronger privacy regimes (e.g., $0.5 \leq \epsilon < 3$). Furthermore, LMO-DP efficiently approximates the sub-optimal DP and fast convergence, compared to the SOTA methods. For instance, fine-tuning RoBERTa-large (with 300M parameters) on the SST-2 dataset can achieve the 92.20% accuracy (given the total privacy budgets $\epsilon = 0.3$ and $\delta = 0$), compared with the ∼50% accuracy of most SOTA methods. We also draw similar findings on text generation tasks while privately fine-tuning GPT-2.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6951