V-FAT: Benchmarking Visual Fidelity Against Text-bias

ACL ARR 2026 January Submission10378 Authors

Keywords: Multimodal Learning, Hallucinations, VQA

Abstract:

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive performance on standard visual reasoning benchmarks. However, there is growing concern that these models rely excessively on linguistic shortcuts rather than genuine visual grounding, a phenomenon we term \textbf{Text Bias}.
In this paper, we investigate the fundamental tension between visual perception and linguistic priors. We decouple the sources of this bias into two dimensions: \textbf{Internal Corpus Bias}, stemming from statistical correlations in pre-training, and \textbf{External Instruction Bias}, arising from the alignment-induced tendency toward sycophancy.
To quantify this effect, we introduce V-FAT (Visual Fidelity Against Text-bias), a diagnostic benchmark comprising 4,026 VQA instances across six semantic domains. V-FAT employs a Three-Level Evaluation Framework that systematically increases the conflict between visual evidence and textual information: (L1) internal bias from atypical images, (L2) external bias from misleading instructions, and (L3) synergistic bias where both coincide. To quantify robustness, we introduce the \textbf{Visual Robustness Score (VRS)}, a metric designed to penalize "lucky" linguistic guesses and reward true visual fidelity. Our evaluation of 12 frontier MLLMs reveals that while models excel in existing benchmarks, they experience significant visual collapse under high linguistic dominance.

Paper Type: Long

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Research Area Keywords: Multimodal Large Language Models, Multimodal Hallucinations

Contribution Types: Model analysis & interpretability, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 10378