MOUCHI: Mitigating Over-forgetting in Unlearning Copyrighted Information
Keywords: machine unlearning, LLM unlearning, derivative knowledge
TL;DR: Propose a novel LLM unlearning framework based on derivative knowledge to avoid over-forgetting
Abstract: Large language models are trained on massive internet datasets, which may inadvertently memorize illegal copyrighted content, making its inclusion unavoidable. Unlearning is a potential solution to remove such content. However, existing unlearning methods often suffer from **over-forgetting**, where the process unintentionally erases knowledge similar to the copyrighted content that falls under fair use and should be preserved. To address this issue, we propose **MOUCHI**, a novel unlearning framework that introduces the concept of **derivative knowledge**, a subset of information derived from copyrighted content that must be retained during unlearning. MOUCHI first generates derivative knowledge and then incorporates a derivative loss function into the unlearning process to mitigate over-forgetting in unlearning copyrighted content. Due to its plug-and-play nature, MOUCHI can be effortlessly integrated into existing unlearning methods. Experimental results show that MOUCHI reduces unintended knowledge loss, improving performance by **up to 145%** compared to baseline methods when evaluated on the derivative set.
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7011
Loading