Not Errors but Guardians: Understanding Sink Tokens in Multimodal LLMs

ICLR 2026 Conference Submission15510 Authors

Keywords: Sink token, Modality bias, Hallucination

Abstract:

Multimodal large language models (MLLMs) achieve remarkable success in vision–language tasks but remain prone to hallucination, often attributed to abnormal attention behaviors.
A recurring phenomenon is the emergence of attention sinks—tokens that absorb large amounts of attention despite limited semantic content.
While previously regarded as artifacts that exacerbate hallucination, we show that in MLLMs certain tokens within system prompts act as stable, system-level attention sinks.
Through causal interventions including masking and content substitution, we find these tokens serve critical functions: anchoring attention to ensure computational stability, influencing outputs, and implicitly tracking the model’s state.
Building on this, we propose the Attention-Budget Hypothesis, which reframes modality bias as a trade-off in attention allocation.
Guided by this perspective, we design SPEAR (Sink-PrEserving Attention Reallocation), an intervention that boosts visual attention while preserving sink functions, achieving effective hallucination mitigation without degrading reasoning.
Our work provides the first systematic characterization of system-level attention sinks in MLLMs and highlights their functional role in both model stability and multimodal reasoning.

Primary Area: interpretability and explainable AI

Submission Number: 15510