Leveraging One-To-Many Relationships in Multimodal Adversarial Defense for Robust Image-Text Retrieval

Futa Kai Waseda, Antonio Tejero-de-Pablos, Isao Echizen

Keywords: Image-Text Retrieval, Adversarial Defense, Vision-Language Model

TL;DR: We are the first to study adversarial defense for VL models in Image-Text Retrieval (ITR). We propose multimodal adversarial training and highlight how leveraging the one-to-many relationship in ITR enhances robustness.

Abstract:

Large pre-trained vision-language models (e.g., CLIP) are vulnerable to adversarial attacks in image-text retrieval (ITR). Existing works primarily focus on defense for image classification, overlooking two key aspects of ITR: multimodal manipulation by attackers, and the one-to-many relationship in ITR, where a single image can have multiple textual descriptions and vice versa (1:N and N:1).
This is the first work that explores defense strategies for robust ITR.
We demonstrate that our proposed multimodal adversarial training, which accounts for multimodal perturbations, significantly improves robustness against multimodal attacks; however, it suffers from overfitting to deterministic one-to-one (1:1) image-text pairs in the training data.
To address this, we conduct a conprehensive study on leveraging one-to-many relationships to enhances robustness, investigating diverse augmentation techniques.
Our findings reveal that diversity and alignment of image-text pairs are crucial for effective defense.
Specifically, text augmentations outperform image augmentations, which tend to create either insufficient diversity or excessive distribution shifts.
Additionally, we find that cross-modal augmentations (e.g., $image \rightarrow text$) can outperform intra-modal augmentations (e.g., $text \rightarrow text$) due to generating well-aligned image-text pairs.
In summary, this work pioneers defense strategies for robust ITR, identifying critical aspects overlooked by prior research, and offers a promising direction for future studies.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7549