Image Triaging for Budget-Aware Universal Attacks on Vision-Language Models

Wei Yong Tan, Dongyue Lu, Wei Tsang Ooi

Keywords: Adversarial Attacks, Universal Adversarial Perturbations, Vision-Language Models, Transferability, Multimodal Robustness, Budgeted Attacks, Attack Efficiency

TL;DR: When attackers can perturb only a small fraction of inputs, selecting the right ones matter. We show that vulnerability-aware triaging using simple image cues significantly improves the efficiency of universal attacks on vision-language models.

Abstract:

Vision-Language Models (VLMs) are typically evaluated under adversarial settings where perturbations can be applied to all inputs. In practice, attackers often face budget constraints, making which inputs to attack as critical as how the perturbation is constructed.
We study budgeted deployment of targeted universal adversarial perturbations (UAPs), where a fixed perturbation must be selectively applied to maximize attack success under limited access. We propose an image-triaging framework that predicts transfer vulnerability from clean, image-level features, trained using surrogate attack outcomes, and requires no model queries at deployment time.
On a road accident monitoring task, across six VLMs and two UAP methods (XTransfer and AnyAttack), our approach substantially improves attack efficiency over random selection, achieving up to 73–97\% ASR at 1\% attack budget. Gains are largest in the low-budget regime, where indiscriminate deployment is highly inefficient. Beyond performance, we find that transfer vulnerability is largely input-intrinsic: simple low-level image statistics capture most of the triaging gains, and models trained with one UAP method generalize to another with minimal degradation. These results suggest that vulnerability signals are shared across attack constructions and can be exploited without access to the victim model.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 98