Differentially Private Synthetic Data: Applied Evaluations and Enhancements

Lucas Rosenblatt, Xiaoyan Liu, Samira Pouyanfar, Eduardo de Leon, Anuj Desai, Joshua Allen

Keywords: privacy, differential privacy, generative adversarial networks, gan, security, synthetic data, evaluation, benchmarking, ensemble

Abstract: Machine learning practitioners frequently seek to leverage the most informative available data, without violating the data owner's privacy, when building predictive models. Differentially private data synthesis protects personal details from exposure, and allows for the training of differentially private machine learning models on privately generated datasets. But how can we effectively assess the efficacy of differentially private synthetic data? In this paper, we survey four differentially private generative adversarial networks for data synthesis. We evaluate each of them at scale on five standard tabular datasets, and in two applied industry scenarios. We benchmark with novel metrics from recent literature and other standard machine learning tools. Our results suggest some synthesizers are more applicable for different privacy budgets, and we further demonstrate complicating domain-based tradeoffs in selecting an approach. We offer experimental learning on applied machine learning scenarios with private internal data to researchers and practitioners alike. In addition, we propose QUAIL, a two model hybrid approach to generating synthetic data. We examine QUAIL's tradeoffs, and note circumstances in which it outperforms baseline differentially private supervised learning models under the same budget constraint.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We present both extensive benchmarking for state-of-the-art differentially private synthesizers and QUAIL, an ensemble-based modeling approach to generating differentially private synthetic data with high utility.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2011.05537/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=BDUtMpMhxa