Latent Diffusion Shield - Mitigating Malicious Use of Diffusion Models Through Latent Space Adversarial Perturbations

Huy Phan, Boshi Huang, Ayush Jaiswal, Ekraam Sabir, Prateek Singhal, Bo Yuan

Abstract: Diffusion models have revolutionized the landscape of generative AI, particularly in the application of text-to-image generation. However, their powerful capability of generating high-fidelity images raises significant security concerns on the malicious use of the state-of-the-art (SOTA) text-to-image diffusion models, notably the risks of misusing personal photos and copyright infringement through the replication of human faces and art styles. Existing pro-tection methods against such threats often suffer from lack of generalization, poor performance, and high computational demands, rendering them unsuitable for real-time or resource-constrained environments. Addressing these chal-lenges, we introduce the Latent Diffusion Shield (LDS), a novel protection approach designed to operate within the latent space of diffusion models, thereby offering robust de-fense against unauthorized diffusion-based image synthesis. We validate LDS's performance through extensive experiments across multiple personalized diffusion models and datasets, establishing new benchmarks in image protection against the malicious use of diffusion models. Notably, the generative version of LDS provides SOTA protection, while being 150 x faster and using 2.6 x less memory.