Why Instruction-Based Unlearning Fails in Diffusion Models?

ACL ARR 2026 January Submission2444 Authors

Keywords: machine unlearning, diffusion model

Abstract: Instruction-based unlearning has proven effective for modifying the behavior of large language models at inference time, but whether this paradigm extends to other generative models remains unclear. In this work, we investigate instruction-based unlearning in diffusion-based image generation models and show, through controlled experiments across multiple concepts and prompt variants, that diffusion models systematically fail to suppress targeted concepts when guided solely by natural-language unlearning instructions. By analyzing both the CLIP text encoder and cross-attention dynamics during the denoising process, we find that unlearning instructions do not induce sustained reductions in attention to the targeted concept tokens, causing the targeted concept representations to persist throughout generation. These results reveal a fundamental limitation of prompt-level instruction in diffusion models and suggest that effective unlearning requires interventions beyond inference-time language control.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: model editing

Contribution Types: Model analysis & interpretability

Languages Studied: english

Submission Number: 2444