The Devil is in the Tokens: Token-Level Structural Analysis Uncovers Hallucinations in LVLMs

ICLR 2026 Conference Submission17344 Authors

Keywords: Large Vision Language Models, Object Hallucination

Abstract: Large vision-language models (LVLMs) achieve strong results in visual reasoning and question answering but remain vulnerable to object hallucination. To explain the causes, prior studies examine global statistics and image-level attention, overlooking the local, and token-level dynamics that underlie hallucinations. We present a fine-grained analysis of hallucination at the patch-token level across layers. Our study reveals two core findings: (i) hallucinated tokens exhibit diffuse, non-localized attention, while faithful tokens show compact attention on relevant patches; and (ii) hallucinated text tokens are not aligned with any object regions. Leveraging these insights, we introduce a lightweight, explainable hallucination detector based on patch-level statistics and hidden features, achieving up to 90% accuracy in token-level hallucination detection. These results demonstrate the value of structural, token-level analysis for understanding and mitigating hallucinations in LVLMs.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 17344