On the Robustness of Latent Diffusion Models

Jianping Zhang, Zhuoer Xu, shiwen cui, Changhua Meng, Weibin Wu, Michael Lyu

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Adversarial Attack, Latent Diffusion Models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Latent diffusion models have achieved state-of-the-art performance on a variety of generative tasks, such as image synthesis and image editing. However, the robustness of latent diffusion models is not well studied. Previous works only focus on the adversarial attacks against the encoder or the output image under white-box settings, regardless of the denoising process. Therefore, in this paper, we aim to analyze the robustness of latent diffusion models more thoroughly. We first study the influence of the components inside latent diffusion models on their white-box robustness. We find out that the denoising process, especially the Resnet, is the most vulnerable to adversarial attacks. In addition to white-box scenarios, we evaluate the black-box robustness of latent diffusion models via transfer attacks, where we consider both prompt-transfer and model-transfer settings and possible defense mechanisms. We conclude that the adversarial vulnerability is inherited with the development of Stable Diffusion models, and the adversarial attacks are still effective when possible defenses are present. Additionally, analyzing the robustness of latent diffusion models needs a comprehensive benchmark dataset, which is missing in the literature. Therefore, to facilitate the research on the robustness of latent diffusion models, we propose two automatic dataset construction pipelines for two kinds of image editing models and release the whole dataset.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3250