Aligning "Hallucinations": Benchmarking LLMs and VLMs with Humans on Blind Visual Question Answering

ACL ARR 2026 January Submission5545 Authors

Keywords: Hallucination, Linguistic Priors, Multimodal Alignment, Visual Question Answering, Explainability

Abstract: Pretrained linguistic knowledge provides an essential semantic foundation for modern Vision-Language Models (VLMs), but its impact on genuine visual grounding remains unclear. We introduce "hallucination alignment", a framework that for the first time systematically compares models and human responses on visual question answering (VQA) without visual input. To this end, we gather the first large human dataset on blinded VQA-derived questions, evaluating Large Language Models (LLMs) and VLMs against human performance and answer patterns. We find that linguistic priors in VLMs enable blind performance exceeding both LLMs and humans. Likewise, answer patterns for both LLMs and VLMs differ significantly from human answers. We show that it is possible to align VLMs to human blinded answers at no cost to visually-grounded inference, creating better aligned multimodal models.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: interpretability, explanation faithfulness, feature attribution, calibration/uncertainty, human-subject application-grounded evaluations, robustness, probing, data shortcuts/artifacts, free-text/natural language explanations

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English, Korean

Submission Number: 5545