Distributional Unlearning: Forgetting Distributions, Not Just Samples

Youssef Allouah, Rachid Guerraoui, Sanmi Koyejo

Keywords: unlearning, GDPR, privacy, sample complexity, machine learning

Abstract: Machine unlearning seeks to remove unwanted information from trained models, initially at the individual‐sample level, but increasingly at the level of entire sub‐populations. In many deployments, models must delete whole topical domains to satisfy privacy, legal, or quality requirements, e.g., removing several users' posts under GDPR or copyrighted web content. Existing unlearning tools remain largely sample-oriented, and straightforward point deletion often leaves enough residual signal for downstream learners to recover the unwanted domain. We introduce distributional unlearning, a data-centric, model-agnostic framework that asks: Given examples from an unwanted distribution and a retained distribution, what is the smallest set of points whose removal makes the edited dataset far from the unwanted domain yet close to the retained one? Using Kullback–Leibler divergence to quantify removal and preservation, we derive the exact Pareto frontier in the Gaussian case and prove that any model retrained on the edited data incurs log-loss shifts bounded by the divergence thresholds. We propose a simple distance-based selection rule satisfying these constraints with a quadratic reduction in deletion budget compared to random removal. Experiments on synthetic Gaussians, Jigsaw Toxic Comments, SMS spam, and CIFAR-10 show 15–72% fewer deletions than random, with negligible impact on retained performance.

Submission Number: 3