Hot PATE: Private Aggregation of Distributions for Diverse Tasks

Edith Cohen, Xin Lyu, Jelani Nelson, Tamas Sarlos, Uri Stemmer

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: PATE, diverse tasks, differential privacy

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Extension of the PATE private learning framework to diverse tasks

Abstract:

The Private Aggregation of Teacher Ensembles (PATE) framework~\cite{PapernotAEGT:ICLR2017} is a versatile approach to privacy-preserving machine learning. In PATE, teacher models are trained on distinct portions of sensitive data, and their predictions are privately aggregated to label new training examples for a student model.
Until now, PATE has primarily been explored with classification-like tasks, where each example possesses a ground-truth label, and knowledge transfer to the student is by labeling random examples.
Generative AI models, however, excel in open ended \emph{diverse} tasks with multiple valid responses and scenarios that may not align with traditional labeled examples.
We propose \emph{hot PATE}, tailored for the diverse setting. In hot PATE, each teacher model contributes a response distribution and the aggregation method must preserve both privacy and diversity of responses. We demonstrate, analytically and empirically, that hot PATE achieves privacy-utility tradeoffs that are comparable to, and in diverse settings, significantly surpass, the baseline ``cold'' PATE.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1642