Teacher Ascent: Robust and Efficient Machine Unlearning via Knowledge Distillation and Continual Learning

ICLR 2026 Conference Submission13702 Authors

Keywords: Machine Unlearning, Teacher Ascent, Catastrophic Forgetting, Continual Learning, Elastic Weight Consolidation, Fine-tuning

TL;DR: This paper introduces Teacher Ascent, a stable and efficient machine unlearning method inspired by continual learning that avoids the catastrophic forgetting that plagues current state-of-the-art approaches like SCRUB+R.

Abstract: Removing specific knowledge from a trained machine learning model is an open problem of increasing importance. Growing dataset sizes increase the likelihood of introducing biased, inaccurate, or private data. Moreover, increasing the number of parameters makes retraining models more costly. While powerful Machine Unlearning methods have emerged as effective alternatives to retraining, their practical application is often hindered by narrow functional ranges for hyperparameters, which typically require access to a retrained model for effective tuning. State-of-the-art methods like SCRUB+R and SSD require precise specification of their hyperparameters to achieve unlearning whilst preventing catastrophic forgetting. We address this challenge by proposing Teacher Ascent (TA), a novel unlearning method that is based on knowledge distillation and continual learning. Inspired by Elastic Weight Consolidation (EWC), TA forgets target data while protecting parameters essential for generalization by using the Fisher Information Matrix. We conduct experiments on MNIST, CIFAR, and Pins Face Recognition across various unlearning scenarios: forgetting entire classes, subclasses, and mislabeled samples. Our results demonstrate that Teacher Ascent both mimics the functional behavior of a retrained model across unlearning tasks while being 6-19 times more efficient than retraining. More importantly, TA mitigates catastrophic forgetting and demonstrates robustness across a wide range of hyperparameters. By overcoming the critical stability and tuning challenges of previous approaches, Teacher Ascent represents a significant step towards making machine unlearning a viable and practical tool for real-world applications.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 13702