Breaking the Gaussian Barrier: Residual-PAC Privacy for Automatic Privatization

Tao Zhang, Yevgeniy Vorobeychik

Abstract:

The Probably Approximately Correct (PAC) Privacy framework [46] provides a powerful instance-based methodology
to preserve privacy in complex data-driven systems. Existing
PAC Privacy algorithms (we call them Auto-PAC) rely on
a Gaussian mutual information upper bound. However, we
show that the upper bound obtained by these algorithms is
tight if and only if the perturbed mechanism output is jointly
Gaussian with independent Gaussian noise. We propose two
approaches for addressing this issue. First, we introduce two
tractable post-processing methods for Auto-PAC, based on
Donsker–Varadhan representation and sliced Wasserstein distances. However, the result still leaves "wasted" privacy budget. To address this issue more fundamentally, we introduce
Residual-PAC (R-PAC) Privacy, an f-divergence-based measure to quantify privacy that remains after adversarial inference. To implement R-PAC Privacy in practice, we propose
a Stackelberg Residual-PAC (SR-PAC) privatization mechanism, a game-theoretic framework that selects optimal noise
distributions through convex bilevel optimization. Our approach achieves efficient privacy budget utilization for arbitrary data distributions and naturally composes when multiple
mechanisms access the dataset. Through extensive experiments, we demonstrate that SR-PAC obtains consistently a
better privacy-utility tradeoff than both PAC and differential
privacy baselines.