The Fundamental Limits of Least-Privilege Learning

Theresa Stadler, Bogdan Kulynych, Nicolas Papernot, Michael Gastpar, Carmela Troncoso

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: societal harms, unintended information leakage

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We formalise the so far only stated informally idea of least-privilege learning and show theoretically and empirically that it often comes with impractical trade-offs.

Abstract: Offloading model training and inference to a service provider is a common practice but raises concerns about data misuse when the provider is untrusted. Collaborative learning and model partitioning aim to address this issue by having clients share a representation of their data instead of the raw data itself. To prevent unintended information leakage, the feature mappings that produce such representations should follow the least-privilege principle, i.e., output representations that are relevant for the intended task, and nothing else. In this work, we provide the first formalisation of the least-privilege principle for machine learning. We first observe that every task comes with fundamental leakage: at the very least, a representation shared for a particular task must reveal the information that can be inferred from the task label itself. Considering this, we formalise the least-privilege principle as a bound on the inference gain about the data behind the representation over what is already revealed through the task’s fundamental leakage. We prove that, under realistic assumptions on the data distribution, there is an inherent trade-off between the utility of representations output by a feature mapping and the leakage of information beyond the intended task. In experiments on image classification, we confirm that any data representation that has good utility for a given prediction task also always leaks more information about the original data than the task label itself. We show that this implies that censoring techniques that hide specific data attributes cannot achieve the desired goal of least-privilege learning.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5214