Geom-Erasing: Geometry-Driven Removal of Implicit Concept in Diffusion Models
Keywords: concept erasure, diffusion model, generative model
Abstract: Fine-tuning diffusion models through personalized datasets is an acknowledged method for improving generation quality across downstream tasks, which, however, often inadvertently generates unintended concepts such as watermarks and QR codes, attributed to the limitations in image sources and collecting methods within specific downstream tasks. Existing solutions suffer from eliminating these unintentionally learned implicit concepts, primarily due to the dependency on the model’s ability to recognize concepts that it actually cannot discern. In this work, we introduce GEOM-ERASING, a novel approach that successfully removes the implicit concepts with either an additional accessible classifier or detector model to encode geometric information of these concepts into the text domain. Moreover, we construct three distinct datasets, each imbued with specific implicit concepts (i.e., watermarks, QR codes, and text) for training and evaluation. Experimental results demonstrate that GEOM-ERASING not only identifies but also proficiently eradicates specific implicit concepts, revealing a significant improvement over the existing methods. The integration of geometric information marks a substantial progression in the precise removal of implicit concepts in diffusion models.
Supplementary Material: pdf
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 152
Loading