Anti-Adapter Armor: A Universal and Authentication-Integrated Framework for Preventing Unauthorized Zero-Shot Image-to-Image Generation

ICLR 2026 Conference Submission17911 Authors

Keywords: copyright protection; data poisoning; diffusion model

Abstract: With the advancement of diffusion models, image generation has entered an era of zero-shot image-to-image synthesis, where highly similar facial identities or artistic styles can be produced using only a single portrait or artwork as input, without requiring any model parameter fine-tuning. However, while these technologies offer significant benefits to artistic creation, they simultaneously introduce non-negligible risks associated with right infringement, such as the unauthorized forgery of facial identities and the plagiarism of artistic styles. To address these risks, this paper proposes Anti-Adapter Armor, the first universal and authentication-integrated framework designed to protect personal images against unauthorized zero-shot image-to-image generation. We begin by analyzing how existing zero-shot image-to-image methods utilize image encoders to convert input images into embeddings, which are injected into the diffusion model's UNet via cross-attention. Based on this, we develop a reversible encryption framework that transforms original image embeddings into diverse encrypted forms based on different passwords. Authorized users can recover the original embeddings using the decryptor and correct passwords for normal image generation. To achieve protection, we propose a multi-targeted adversarial attack that transfers the original image embeddings into the encrypted forms by adding adversarial perturbation. Therefore, the protected images are equipped with a protective coating that restricts unauthorized users to generating encrypted content exclusively. Extensive experiments show that our approach outperforms state-of-the-art protection methods in preventing unauthorized zero-shot image-to-image generation, while enabling adaptable and secure authentication for authorized users.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 17911