Machine Unlearning Meets Continuous Learning: A Theoretical Analysis

Yiting Hu, Lingjie Duan

Keywords: continual learning; certified machine unlearning; excess risk; unlearning loss

Abstract: Machine unlearning is designed to remove specific data from a trained model to protect privacy. However, a significant challenge arises in the field of continual learning, where models evolve without full access to past data due to ever-increasing storage burdens and environmental constraints. This is because current certified unlearning algorithms do not accommodate the complex model evolution in continual learning. In this work, we establish the first theoretical foundation connecting continual learning and machine unlearning, where the former aims to preserve knowledge across all previously trained tasks, while the latter requires efficient and immediate forgetting upon receiving unlearning requests. We successfully adapt two popular certified unlearning approaches, one leveraging gradients and the other Hessians, to function within a continual learning framework. We provide theoretical performance guarantees by analyzing two key metrics: the excess risk under continual learning and the unlearning loss. The combination of these two metrics jointly determines the final post-unlearning excess risk. Our analysis shows that our Hessian-based adaption algorithm largely outperforms the gradient-based algorithm, while the latter offers an advantage by reducing the storage cost to zero. We validate these theoretical findings with experiments on the MNIST dataset, which also demonstrate the effect of the sequence of unlearning requests.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 6890