Machine Unlearning in Low-dimensional Feature Subspace

Kun Fang, Qinghua Tao, Junxu Liu, Yaxin Xiao, Qingqing Ye, Jian Sun, Haibo Hu

Keywords: machine unlearning, low-dimensional subspace

Abstract:

Data privacy in modern neural networks attracts growing interest in Machine Unlearning (MU), which aims at removing the knowledge of particular data from a pretrained model and meanwhile maintaining good performances on the remaining data.
In this work, a new perspective upon low-dimensional feature subspaces is presented to investigate MU.
We firstly demonstrate the potentials of separating the remaining and forgetting data in a low-dimensional feature subspace.
Then, such separability motivates us to seek a subspace ${\tt range}(\bf U)$ on the features of the pretrained model for unlearning, where the information of the remaining data is preserved and that of the forgetting data is therein diminished, leading to the proposed new method named SUbspace UNlearning (SUN).
Compared to mainstream MU methods that require direct and massive access to the training data for model updating, SUN offers two key advantages well resolving these significant challenges in practice.
(i) SUN avoids frequent data visits and optimizes $\bf U$ involving two covariance matrices, which only requires one-shot feature fetching and thereby alleviates data privacy risks and computation.
(ii) SUN in implementation simply serves as a plug-in module to the pretrained model without modifications to its original parameters, reducing the parameter number and computational overhead by orders of magnitude, which is of great practicality for handling multiple unlearning requests.
Extensive numerical experiments verify our superior unlearning accuracy with significantly less parameters and computing time over variants of models, datasets, tasks, and applications.
Code is available at the anonymous link https://anonymous.4open.science/r/4352/.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 4352