Localized-Attention-Guided Concept Erasure for Text-to-Image Diffusion Models

Zhuan Shi, Alireza Dehghanpour Farashah, Rik de Vries, Golnoosh Farnadi

Track: Regular paper

Keywords: Machine Unlearning, Text-to-image diffusion model, Concept Erasure

Abstract: Concept erasure has become a fundamental safety requirement for text‑to‑image diffusion models, enabling removal of objectionable or copyrighted content without costly retraining. To preserve generative capacity, localized concept erasure is proposed which confines edits to the region occupied by the target concept and leaves the remainder of the scene untouched. However, existing localized concept erasure still suffer from a Concept Neighborhood gap: suppressing the target often attenuates neighboring, semantically related concepts, diminishing overall fidelity and limiting practical utility. To bridge this gap, we present Localized‑Attention‑Guided Concept Erasure (LACE), a training‑free framework whose three stages progress from coarse to fine control: (1) Representation‑space projection, which suppresses the target concept subspace while reinforcing semantic neighbors; (2) Attention‑guided spatial gate, which derives a spatial mask identifying regions of residual concept activation and conduct attention suppression; (3) Gated Feature Clean-up, which performs a hard scrub on gated feature activations. This three-stage pipeline enables precise and localized removal of visual concepts while retaining semantic structure and expressiveness. Experiments show that LACE effectively removes targeted concepts, preserves semantically related neighbors, and maintains overall image composition.

Submission Number: 37