V$^2$R-Bench: Holistically Evaluating LVLM Robustness to Fundamental Visual Variations

ACL ARR 2025 February Submission5026 Authors

Abstract:

Large Vision Language Models (LVLMs) have shown impressive performance on various vision-language tasks.
However, while objects in natural scenes inevitably exhibit visual variations in position, scale, orientation, and context due to changes in viewpoint and environment, the robustness of LVLMs to these fundamental visual variations remains largely unexplored.
To address this gap, we introduce V$^2$R-Bench, a comprehensive benchmark framework for evaluating Visual Variation Robustness of LVLMs, which encompasses automated evaluation dataset generation and principled metrics for thorough robustness assessment.
Through extensive evaluation of 13 LVLMs, we reveal a surprising vulnerability to visual variations, affecting even advanced models that excel at complex vision-language tasks yet significantly underperform on simple tasks like object recognition. Interestingly, these models exhibit a distinct visual position bias that contradicts theories of effective receptive fields and demonstrate a human-like visual acuity threshold.
To identify the source of these vulnerabilities, we propose a systematic framework for component-level analysis, featuring a novel visualization approach for aligned visual features. Results show that these vulnerabilities stem from error accumulation in the pipeline architecture and inadequate multimodal alignment. Complementary experiments with synthetic data further demonstrate that these limitations are fundamentally architectural challenges, underscoring the need for architectural innovations in future LVLM designs.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: multimodal alignment

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Data resources

Languages Studied: English

Submission Number: 5026