SELF-SUPERVISED PRETRAINING FOR DIFFERENTIALLY PRIVATE LEARNING

Arash Asadian, Evan Weidner, Lei Jiang

Keywords: differential privacy, contrastive learning, learned features, one image

TL;DR: We demonstrate self-supervised pretraining is a scalable solution to deep learning with differential privacy regardless of the size of available public datasets in image classification.

Abstract: We demonstrate self-supervised pretraining (SSP) is a scalable solution to deep learning with differential privacy (DP) regardless of the size of available public datasets in image classification. When facing the lack of public datasets, we show the features generated by SSP on only one single image enable a private classifier to obtain a much better utility than the non-learned handcrafted features under the same privacy budget. When a moderate or large size public dataset is available, the features produced by SSP greatly outperform the features trained with labels on various complex private datasets under the same private budget. We also compared multiple DP-enabled training frameworks to train a private classifier on the features generated by SSP.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Social Aspects of Machine Learning (eg, AI safety, fairness, privacy, interpretability, human-AI interaction, ethics)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/self-supervised-pretraining-for/code)