Breaking Weight Entanglement: Machine Unlearning with Nonlinearity

Yingdan Shi, Ren Wang

Keywords: machine unlearning, mode connectivity, model editing

Abstract: Machine Unlearning (MU) seeks to eliminate the influence of specific training data from a pretrained model. One existing approach achieves unlearning via linear parameter updates by task arithmetic. However, linear editing parameters often suffers from the weight entanglement issue. In this work, we introduce an unlearning framework Mode Connectivity Unlearning (MCU), that leverages mode connectivity to discover a nonlinear unlearning pathway in parameter space. To boost both effectiveness and efficiency, we further incorporate a parameter masking strategy that enhances the forgetting process while reducing computational costs. Additionally, we present an adaptive mechanism for our unlearning penalty coefficient, which adaptively balances forgetting quality and model utility without the need for manual hyperparameter search. Distinct from existing MU techniques that produce a single unlearning model, MCU reveals multiple unlearning models along the pathway. Overall, MCU functions as a plug-and-play framework that can be integrated into all existing MU methods, consistently enhancing their unlearning performance.

Submission Number: 9