PAIR: Pre-denosing Augmented Image Retrieval Model for Defending Adversarial Patches

Ziyang Zhou, Pinghui Wang, Zi Liang, Ruofei Zhang, Haitao Bai

Abstract: Deep neural networks are widely used in retrieval systems. However, they are notoriously vulnerable to attack. Among the various forms of adversarial attacks, the patch attack is one of the most threatening forms. This type of attack can introduce cognitive biases into the retrieval system by inserting deceptive patches into images. Despite the seriousness of this threat, there are still no well-established solutions for image retrieval systems. In this paper, we propose the Pre-denosing Augmented Image Retrieval (PAIR) model, a new approach designed to protect image retrieval systems against adversarial patch attacks. The core strategy of PAIR is to dynamically and randomly reconstruct entire images based on their semantic content. This purifies well-designed patch attacks while preserving the semantic integrity of the images. Furthermore, we present a novel training strategy that incorporates a semantic discriminator. This discriminator significantly improves PAIR's ability to capture real semantics and reconstruct images. Experiments show that PAIR significantly outperforms existing defense methods. It effectively reduces the success rate of two state-of-the-art patch attack methods to below 5\%, achieving a 14\% improvement over current leading methods. Moreover, in defending against other forms of attack, such as global perturbation attacks, PAIR also achieves competitive results. The codes are available at: https://anonymous.4open.science/r/PAIR-8FD2.

Primary Subject Area: [Engagement] Multimedia Search and Recommendation

Secondary Subject Area: [Content] Media Interpretation, [Content] Vision and Language

Relevance To Conference:

1. The problem solved in this paper is defending against patch attacks in multimodal retrieval, where both image and text modalities are involved. An attacker can insert an attack patch into the image to affect the retrieval model's perception and understanding of the image. This attack causes the users to see the retrieval results that the attacker wants them to see.

2. Our proposed PAIR method is the first defense model against patch attacks in multimodal retrieval. PAIR destroys attack patches by denoising and recovering images while preserving their semantic information. Therefore, it is compatible with most downstream retrieval models.

3. We design a new training method that utilizes a semantic discriminator to improve the PAIR’s performance. The defense model performs similarly to the original model on clean samples and has stronger defenses.

4. Experiments demonstrate that PAIR is highly effective at defending two state-of-the-art attack methods. It reduces the success rate of attacks to less than 5%.

5. Furthermore, PAIR demonstrates competitive results in defending against other forms of attacks, such as global perturbation attacks.

Submission Number: 3752