MetaCloak: Preventing Unauthorized Subject-driven Text-to-image Synthesis Via Meta-learning

Yixin Liu, Chenrui Fan, Yutong Dai, Xun Chen, Pan Zhou, Lichao Sun

Keywords: Poisoning Attack, Unauthorized Synthesis, DreamBooth, Text-to-Image Diffusion Models

TL;DR: We propose MetaCloak, a robust solution utilizing meta-learning and transformation-robust noise crafting to prevent unauthorized, subject-driven text-to-image synthesis by DreamBooth models.

Abstract:

Text-to-image diffusion models, epitomized by DreamBooth, allow seamless generation of personalized images from scant reference photos. Yet, these tools, in the wrong hands, can fabricate misleading or harmful content, endangering individuals. To address this, existing poisoning-based approaches perturb user images in an imperceptible way to render them "unlearnable" from malicious uses. We identify two limitations of these defending approaches: i) sub-optimal due to the hand-crafted heuristics for solving intractable bilevel optimization; and ii) lack of robustness against simple countermeasures like Gaussian filtering transformations. To solve these challenges, we propose MetaCloak to prevent the unauthorized subject-driven text-to-image synthesis of DreamBooth finetuning.
MetaCloak combines a first-order method that approximately solves the bilevel problem via meta-learning and a transformation-robust noise crafting process. Specifically, MetaCloak unrolls the training trajectory of the inner optimization loop and conducts iterative updates between surrogate models and the perturbation. To improve the robustness and transferability of our perturbation across models, we further propose \textit{curricular ensembling} by looping over steps-staggered clean surrogate diffusion models of different versions.
Furthermore, to bypass transformation defenses, MetaCloak crafts transformation-robust perturbation by conducting denoising-error maximization for semantic distortion. Extensive experiments on the VGGFace2 and CelebA-HQ datasets show that MetaCloak significantly outperforms existing attacking approaches. Notably, MetaCloak can successfully fool several online DreamBooth training services like Replicate in a black-box manner, demonstrating the defense effectiveness of MetaCloak in real-world scenarios.

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3609