OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models

ACL ARR 2025 May Submission2354 Authors

Abstract:

Large language models (LLMs) trained over extensive corpora risk memorizing sensitive, copyrighted, or toxic content.
To address this, we propose \textbf{OBLIVIATE}, a robust unlearning framework that removes targeted data while preserving model utility.
The framework follows a structured process: extracting target tokens, building retain sets, and fine-tuning with a tailored loss function comprising three components---masking, distillation, and world fact.
Using low-rank adapters (LoRA), it ensures efficiency without compromising unlearning quality.
We conduct experiments on multiple datasets, including the Harry Potter series, WMDP, and TOFU, using a comprehensive suite of metrics: \emph{forget quality} (new document-level memorization score), \emph{model utility}, and \emph{fluency}.
Results demonstrate its effectiveness in resisting membership inference attacks, minimizing the impact on retained data, and maintaining robustness across diverse scenarios.

Paper Type: Long

Research Area: Ethics, Bias, and Fairness

Research Area Keywords: Unlearning, Large Language Models

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English

Submission Number: 2354