Distributed PATE and CaPC on a DIET: Private Knowledge Transfer without Public Data or Private Inference

Michel Meintz, Adam Dziedzic, Franziska Boenisch

Keywords: PATE, programmatically generated data, data free knowledge distillation, CaPC, privacy, differential privacy

TL;DR: We present a distributed private learning approach that eliminates both PATE's requirement for in-distribution public data and CaPC's reliance on costly private inference, using programmatically generated data and data-free distillation.

Abstract: The PATE algorithm is one of the canonical approaches to private machine learning. It leverages a private dataset to label a public dataset, enabling knowledge transfer from teachers to a student model under differential privacy (DP) guarantees. However, PATE's reliance on public data from the same distribution as the private data poses a fundamental limitation, particularly in domains such as healthcare and finance, where in-distribution public data is typically unavailable. In this work, we propose DIET-PATE which overcomes this limitation. Therefore, it combines programmatically generated data and data-free knowledge distillation. Our experiments demonstrate that DIET-PATE closely matches the performance of standard PATE, despite the absence of in-distribution public data. Furthermore, we show that our approach seamlessly extends to distributed collaborative learning with CaPC. In this setting, only PATE-based learning can be used to provide DP guarantees, as teacher models are trained by different entities and exchange knowledge solely via labels. By eliminating the need for in-distribution data during knowledge transfer, our method removes CaPC’s reliance on private inference with encrypted data, substantially reducing computational overhead and, for the first time, enabling the use of more complex models and learning tasks. Moreover, leveraging programmatically generated data allows parties in CaPC to jointly train a global model, rather than just improving local ones, thereby achieving significantly higher utility. These advances extend the practicality of distributed private learning with PATE and CaPC to sensitive and complex domains.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 1685