Data-Centric Unlearning: Optimizing Labels and Retain Data via Learning Dynamics

ICLR 2026 Conference Submission121 Authors

Keywords: Machine unlearning; learning dynamics; data optimization; data selection

TL;DR: Our work optimizes key training data (labels, retain data) in machine unlearning via learning dynamics to boost SOTA for classifiers/LLMs.

Abstract: Machine unlearning mitigates adverse effects from erroneous, outdated, or private training data. Although unlearning algorithms have advanced for classifiers and LLMs, the critical role of unlearning training data quality remains largely unexplored. This work addresses this fundamental gap by systematically investigating how to construct effective unlearning training sets, focusing on optimal label assignment for samples and strategic selection for the retain set. We leverage learning dynamics theory to analyze the impact of training data on unlearning performance. Precisely, we derive: (1) an optimal label assignment scheme for both unlearning and retain samples, and (2) the principle that neighborhood and boundary samples are most beneficial for inclusion in the retain set. We translate these theoretical insights into data optimization algorithms tailored for both classifiers and LLMs unlearning. Extensive experiments across classifier and LLMs unlearning tasks demonstrate that our data optimization strategies significantly enhance the performance of existing SOTA unlearning algorithms. Our work establishes data optimization as a crucial pillar for effective machine unlearning.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 121