Enhancing Adversarial Transferability in Vision-Language Models via Search-Space Expansion

Peng-Fei Zhang, Zixin Wang, Zi Huang

Keywords: Adversarial Attack, Vision-Language Models, Robustness

TL;DR: This paper propose a novel transfer-based attack to evaluate the robustness and security of VLP models across different scenarios

Abstract: Adversarial attacks are crucial for evaluating the robustness of vision-language pre-trained (VLP) models. However, existing methods suffer from limited transferability across unseen models, limiting their effectiveness as a universal robustness probe. We attribute this partially to the narrow search space of adversarial examples, which can trap optimization in local optima and lead to overfitting. To address this, we propose SEA (\textbf{S}earch-space \textbf{E}xpansion \textbf{A}ttack), a unified framework that improves cross-model transferability by enlarging the adversarial search space across both modalities. For images, SEA leverages historical updates to explore novel optimization directions, effectively avoiding suboptimal optimization trajectories and overfitting. For text, SEA considers both individual word importance and word interactions, recognizing that less salient words can sometimes yield stronger and more transferable attacks. It performs word substitutions across multiple influential positions rather than focusing solely on the most salient word. Consequently, SEA can substantially disrupt cross-modal interactions across different models. Extensive experiments on diverse benchmarks, VLP models and tasks, supported by rigorous theoretical analysis, demonstrate that SEA significantly advances the state of the art. The source code is provided in the supplementary material.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 5303