Sharpness-Aware Machine Unlearning

Haoran Tang, Rajiv Khanna

Abstract:

We characterize the effectiveness of Sharpness-aware minimization (SAM) under
machine unlearning scheme, where unlearning forget signals interferes with learning retain signals. While previous work prove that SAM improves generalization
with noise memorization prevention, we show that SAM abandons such denoising
property when fitting the forget set, leading to various test error bounds depending
on signal strength. We further characterize the signal surplus of SAM in the order
of signal strength, which enables learning from less retain signals to maintain
model performance and putting more weight on unlearning the forget set. Empirical studies show that SAM outperforms SGD with relaxed requirement for retain
signals and can enhance various unlearning methods either as pretrain or unlearn
algorithm. Observing that overfitting can benefit more stringent sample-specific
unlearning, we propose Sharp MinMax, which splits the model into two to learn
retain signals with SAM and unlearn forget signals with sharpness maximization,
achieving best performance. Extensive experiments show that SAM enhances
unlearning across varying difficulties measured by data memorization, yielding
decreased feature entanglement between retain and forget sets, stronger resistance
to membership inference attacks, and a flatter loss landscape.