Inversion-Guided Weight Masking and Patching: Purification-Free Defense for Diffusion Models against Adversarial Perturbation

ICLR 2026 Conference Submission20955 Authors

Keywords: adversarial robustness, customized diffusion model, purification-free defense, adversarial perturbation

TL;DR: We introduce IMAP, a purification-free defense against adversarial perturbations for diffusion models using DDIM inversion-guided weight masking and patching; it improves generation quality without clean data, validated by extensive experiments.

Abstract:

Diffusion models have demonstrated outstanding generative capabilities but remain vulnerable to adversarial perturbations. These perturbations, originally intended for data copyright protection, expose a critical robustness weakness in diffusion models. Existing defenses mainly based on data purification, which require prior assumptions and incur high computational costs.
In this work, we investigate the impact of perturbations on different modules of diffusion models and introduce Inversion-Guided Weight Masking and Patching (IMAP), a purification-free method designed to restore diffusion models customized on adversarially perturbed data. Our approach first applies a prompt re-mapping strategy before customization. We then use DDIM inversion to identify critical convolutional kernels affected by perturbations and perform weight masking and adaptive patching to restore the model.
IMAP requires no clean data or costly per-image purification. Extensive experiments on CelebA-HQ and VGGFace2 demonstrate that IMAP significantly improves generation quality under various adversarial scenarios.
Furthermore, through comparisons with purification-based techniques, we demonstrate the effectiveness of IMAP and show that it can be effectively integrated with existing purification-based methods.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 20955