Noise-Aware Differentially Private Regression via Meta-Learning

Ossi Räisä, Stratis Markou, Matthew Ashman, Wessel P Bruinsma, Marlon Tobaben, Antti Honkela, Richard E. Turner

Keywords: differential privacy, meta-learning, neural processes, Gaussian processes, sim-to-real, probabilistic regression

TL;DR: We train a neural network that can output a differentially private probabilistic regression model from a data set in one forward pass.

Abstract: Many high-stakes applications require machine learning models that protect user privacy and provide well-calibrated, accurate predictions. While Differential Privacy (DP) is the gold standard for protecting user privacy, standard DP mechanisms typically significantly impair performance. One approach to mitigating this issue is pre-training models on simulated data before DP learning on the private data. In this work we go a step further, using simulated data to train a meta-learning model that combines the Convolutional Conditional Neural Process (ConvCNP) with an improved functional DP mechanism of Hall et al. (2013), yielding the DPConvCNP. DPConvCNP learns from simulated data how to map private data to a DP predictive model in one forward pass, and then provides accurate, well-calibrated predictions. We compare DPConvCNP with a DP Gaussian Process (GP) baseline with carefully tuned hyperparameters. The DPConvCNP outperforms the GP baseline, especially on non-Gaussian data, yet is much faster at test time and requires less tuning.

Primary Area: Privacy

Submission Number: 6835