SPARTA: Spectral Prompt Agnostic Adversarial Attack on Medical Vision-Language Models

Asif Hanif, Muhammad Zaigham Zaheer, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer

Keywords: Vision-Language Models · Adversarial Attack · Transferability · Spectral Attack · Visual Question Answering

Abstract:

Medical Vision-Language Models (Med-VLMs) are gaining
popularity in different medical tasks, such as visual question-answering
(VQA), captioning, and diagnosis support. However, despite their impressive
performance, Med-VLMs remain vulnerable to adversarial attacks,
much like their general-purpose counterparts. In this work, we investigate
the cross-prompt transferability of adversarial attacks on Med-
VLMs in the context of VQA. To this end, we propose a novel adversarial
attack algorithm that operates in the frequency domain of images and
employs a learnable text context within a max-min competitive optimization
framework, enabling the generation of adversarial perturbations that
are transferable across diverse prompts. Evaluation on three Med-VLMs
and four Med-VQA datasets shows that our approach outperforms the
baseline, achieving an average attack success rate of 67% (compared to
baseline’s 62%).

Submission Number: 7