Communication Efficient Federated Learning with Secure Aggregation and Differential Privacy

Wei-Ning Chen, Christopher A. Choquette-Choo, Peter Kairouz

Keywords: machine learning, federated learning, differential privacy, distributed differential privacy, secure aggregation, multi party computation, linear compression, sparsity

TL;DR: We present linear compression schemes for federated learning with secure aggregation and differential privacy, and show that we can attain compression rates of up to 50x with no significant decrease in converged model test accuracy.

Abstract: Optimizing the \puc tradeoff is a key challenge for federated learning. Under distributed differential privacy (DP) via secure aggregation (SecAgg), we prove that the worst-case communication cost per client must be at least $\Omega\left( d \log \left( \frac{n^2\varepsilon^2}{d} \right) \right)$ to achieve $O\left( \frac{d}{n^2\varepsilon^2} \right)$ centralized error, which matches the error under central DP. Despite this bound, we leverage the near-sparse structure of model updates, evidenced through recent empirical studies, to obtain improved tradeoffs for distributed \DP. In particular, we leverage linear compression methods, namely sketching, to attain compression rates of up to $50\times$ with no significant decrease in model test accuracy achieving a noise multiplier $0.5$. Our work demonstrates that fundamental tradeoffs in differentially private federated learning can be drastically improved in practice.

Paper Under Submission: The paper is NOT under submission at NeurIPS