On Optimization Complexity of Second-Order Certified Unlearning

Nikita Doikov, Anastasia Koloskova

Keywords: Certified Unlearning, Anisotropic Gaussian Mechanism, Second-Order Methods, Complexity of Unlearning

TL;DR: A second-order unlearning algorithm with provable benefits over first-order methods and state-of-the-art global convergence

Abstract:

We study machine unlearning: the removal of
memorized training data from a trained model.
Specifically, we investigate the algorithmic complexity
of certified unlearning from an optimization
perspective. We formalize the goal of an
unlearning algorithm as simultaneously achieving
certified unlearning and optimization accuracy.
Utilizing the notion of uniformly convex regularizers,
we prove new bounds on the distance between
initial and unlearned models using a novel
substitute for generalization error. Thus we theoretically
demonstrate that if the removed data is
well-predicted by the unlearned model, the corresponding
optimization problem is simple. Furthermore,
we develop a new second-order unlearning
algorithm with an anisotropic Gaussian mechanism
and state-of-the-art global convergence. We
prove fast rates for our method in achieving certified
unlearning for linear models with quasi-selfconcordant
losses. As a direct application, our
theory covers unlearning for logistic and exponential
regressions and shows a provable benefit of
utilizing second-order information compared to
first-order unlearning methods.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 48