From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearning

ACL ARR 2026 January Submission591 Authors

Keywords: Unlearning, Large Language Models

Abstract:

Although machine unlearning is essential for removing private, harmful, or copyrighted content from LLMs, current benchmarks often fail to faithfully represent the true ``forgetting scope'' learned by the model.
We formalize two distinct unlearning granularities, domain-level and instance-level, and propose \BiForget, an automated framework for synthesizing high-quality forget sets.
Unlike prior work relying on \emph{external} generators, \BiForget exploits the target model per se to elicit data that matches its internal knowledge distribution through seed-guided and adversarial prompting.
Our experiments across diverse benchmarks show that it achieves a superior balance of relevance, diversity, and efficiency.
Quantitatively, in the Harry Potter domain, it improves relevance by ${\sim}20$ and diversity by ${\sim}$0.05 while \emph{halving} the total data size compared to SOTAs.
Ultimately, it facilitates more robust forgetting and better utility preservation, providing a more rigorous foundation for evaluating LLM unlearning.

Paper Type: Long

Research Area: Ethics, Bias, and Fairness

Research Area Keywords: Unlearning, Large Language Models

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English

Submission Number: 591