Avoiding Copyright Infringement via Machine Unlearning
Abstract: Pre-trained Large Language Models (LLMs) have demonstrated remarkable capabilities but also pose risks by learning and generating copyrighted material, leading to significant legal and ethical concerns. To address these issues, it is critical for model owners to be able to unlearn copyrighted content at various time steps. We explore the setting of sequential unlearning, where copyrighted content is removed over multiple time steps—a scenario that has not been rigorously addressed. To tackle this challenge, we propose Stable Sequential Unlearning (SSU), a novel unlearning framework for LLMs, designed to have a more stable process to remove copyrighted content from LLMs throughout different time steps using task vectors, by incorporating additional random labeling loss and applying gradient-based weight saliency mapping. Experiments demonstrate that SSU finds a good balance between unlearning efficacy and maintaining model's general knowledge compared to existing baselines.
Paper Type: Long
Research Area: Machine Learning for NLP
Research Area Keywords: Machine Unlearning, Large Language Models, Trustworthy ML,
Contribution Types: NLP engineering experiment
Languages Studied: English
Submission Number: 4040
Loading