Differentially Private Federated Bayesian Optimization with Distributed Exploration

Zhongxiang Dai, Bryan Kian Hsiang Low, Patrick Jaillet

Keywords: Bayesian Optimization, Federated Learning, Federated Hyperparameter Tuning, Differential Privacy, Distributed Exploration

Abstract: Bayesian optimization (BO) has recently been extended to the federated learning (FL) setting by the federated Thompson sampling (FTS) algorithm, which has promising applications such as federated hyperparameter tuning. However, FTS is not equipped with a rigorous privacy guarantee which is an important consideration in FL. Recent works have incorporated differential privacy (DP) into the training of deep neural networks through a general framework for adding DP to iterative algorithms. Following this general DP framework, our work here integrates DP into FTS to preserve user-level privacy. We also leverage the ability of this general DP framework to handle different parameter vectors, as well as the technique of local modeling for BO, to further improve the utility of our algorithm through distributed exploration (DE). The resulting differentially private FTS with DE (DP-FTS-DE) algorithm is endowed with theoretical guarantees for both the privacy and utility and is amenable to interesting theoretical insights about the privacy-utility trade-off. We also use real-world experiments to show that DP-FTS-DE achieves high utility (competitive performance) with a strong privacy guarantee (small privacy loss) and induces a trade-off between privacy and utility.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

TL;DR: We introduce the first federated Bayesian optimization algorithm with a rigorous privacy guarantee, and we demonstrate the privacy-utility trade-off both in theory and in practice.

Code: https://github.com/daizhongxiang/Differentially-Private-Federated-Bayesian-Optimization