Scaling Private Deep Learning with Opacus: Advances for Large Language Models

Sai Aparna Aketi, Will Bullock, Iden Kalemaj, Enayat Ullah, Huanyu Zhang

Keywords: Differential Privacy, DP-SGD, LLM, Pytorch

TL;DR: An overview of the Differentially Private SGD (DP-SGD) library built on PyTorch, along with the challenges and solutions related to large language models (LLMs).

Abstract: We introduce Library X, a free, open-source PyTorch library for training deep learning models with differential privacy. Library X is designed for simplicity, flexibility, and speed. It provides a simple and user-friendly API, and enables machine learning practitioners to make a training pipeline private by adding as little as two lines to their code. In this paper, we first provide a brief overview of Library X, and then reveal the challenges posed by the prevalence of large language models (LLMs). To tackle these challenges, we propose several new features, either recently added or planned for the next version, including Fast Gradient and Ghost Clipping, model parallelism, parameter-efficient fine-tuning (PEFT), and mixed precision training.

Submission Number: 24