When to Forget? Complexity Trade-offs in Machine Unlearning

Martin Van Waerebeke, Marco Lorenzi, Giovanni Neglia, Kevin Scaman

Keywords: Machine Unlearning, Theory, Differential Privacy, Optimization

TL;DR: We show when first-order unlearning can be much faster than retraining, and when it cannot.

Abstract:

Machine Unlearning (MU) aims at removing the influence of specific data points from a trained model at a fraction of the cost of full model retraining. In this paper, we analyze the efficiency of unlearning methods and establish the first upper and lower bounds on minimax computation times for this problem, characterizing the performance of the most efficient algorithm against the most difficult objective function. Specifically, for strongly convex objective functions and under the assumption that the forget data is inaccessible to the unlearning method, we provide a phase diagram for the unlearning complexity ratio---a novel metric that compares the computational cost of the best unlearning method to full model retraining. The phase diagram shows three regimes: one where unlearning is too costly, one where it’s trivial, and one where it outperforms retraining. These findings highlight the critical role of factors such as data dimensionality, the number of samples to forget, and privacy constraints in determining the practical feasibility of unlearning.

Submission Number: 2