Evaluating Vulnerabilities in Vision-Language Models: Impact of Behavior-Induced Interference

Yuwei Chen, Shiyong Chu

Keywords: Vision-Language Models, Adversarial Robustness, Behavioral Perturbations, Decision Drift, Priority Inversion, Adversarial Vulnerability

Abstract: Vision-language models (VLMs) are increasingly recognized as central components for multimodal scene understanding and decision-support in complex autonomous systems. While existing research on VLM robustness has predominantly focused on digital-domain perturbations such as pixel-level noise or adversarial prompt injection, adversarial influence in dynamic environments can also manifest through structured changes in observable behavior. In this work, we investigate the susceptibility of VLMs to behavior-induced perturbations within system dynamics, rather than the sensor data stream. Leveraging a high-fidelity simulation environment, we design a series of behavior-level variations, including motion pattern changes, altitude adjustments, and emission reconfiguration, that preserve platform identity and task semantics while altering observable action patterns. To quantify decision instability, we introduce two metrics: Ranking Drift, which measures overall shifts in priority ranking, and Priority Inversion Rate, which captures cases where the highest-priority entity is erroneously deprioritized. Evaluations across multiple state-of-the-art multimodal foundation models demonstrate that even semantically invariant behavioral variations can induce significant Ranking Drift and frequent priority inversion. Our results reveal a critical and underexplored vulnerability: behavioral salience alone can systematically bias prioritization in multimodal reasoning pipelines. These findings highlight the urgent need for robustness evaluations that consider behavior-level dynamics when deploying VLMs in safety-critical autonomous systems.

Submission Number: 12