Scaling Laws for Differentially Private Language Models

Ryan McKenna, Yangsibo Huang, Amer Sinha, Borja Balle, Zachary Charles, Christopher A. Choquette-Choo, Badih Ghazi, Georgios Kaissis, Ravi Kumar, Ruibo Liu, Da Yu, Chiyuan Zhang

TL;DR: We aim to understand and quantify the compute-privacy-utility trade-offs in language models training

Abstract: Scaling laws have emerged as important components of large language model (LLM) training as they can predict performance gains through scale, and provide guidance on important hyper-parameter choices that would otherwise be expensive. LLMs also rely on large, high-quality training datasets, like those sourced from (sometimes sensitive) user data. Training models on this sensitive user data requires careful privacy protections like differential privacy (DP). However, the dynamics of DP training are significantly different, and consequently their scaling laws are not yet fully understood. In this work, we establish scaling laws that accurately model the intricacies of DP LLM training, providing a complete picture of the compute-privacy-utility and the optimal training configurations in many settings.

Lay Summary: This paper systematically studies the privacy / utility / compute trade-off for training language models with DP-SGD. Our experiments and analysis allow training compute-optimal language models by efficiently allocating the compute budget among the batch size, model size, and number of iterations. The results cover an exhaustive range of reasonable privacy budgets and dataset sizes, both of which significantly influence the optimal compute allocation and achievable loss.

Link To Code: https://github.com/google-research/google-research/tree/master/dp_scaling_laws

Primary Area: Social Aspects->Privacy

Keywords: scaling laws, differential privacy

Submission Number: 14196