Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation

Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, Heng Tao Shen

Abstract:

Despite their success, Large Vision-Language Models
(LVLMs) remain vulnerable to hallucinations. While existing
studies attribute the cause of hallucinations to insufficient vi-
sual attention to image tokens, our findings indicate that hal-
lucinations also arise from interference from instruction to-
kens during decoding. Intuitively, certain instruction tokens
continuously distort LVLMs’ visual perception during decod-
ing, hijacking their visual attention toward less discrimina-
tive visual regions. This distortion prevents them integrating
broader contextual information from images, ultimately lead-
ing to hallucinations. We term this phenomenon “Attention
Hijacking”, where disruptive instruction tokens act as “At-
tention Hijackers”. To address this, we propose a novel,
training-free strategy namely Attention HIjackers Detection
and Disentanglement (AID), designed to isolate the influ-
ence of Hijackers, enabling LVLMs to rely on their context-
aware intrinsic attention map. Specifically, AID consists of
three components: First, Attention Hijackers Detection iden-
tifies Attention Hijackers by calculating instruction-driven vi-
sual salience. Next, Attention Disentanglement mechanism
is proposed to mask the visual attention of these identified
Hijackers, and thereby mitigate their disruptive influence
on subsequent tokens. Finally, Re-Disentanglement recal-
culates the balance between instruction-driven and image-
driven visual salience to avoid over-masking effects. Exten-
sive experiments demonstrate that AID significantly reduces
hallucination across various LVLMs on several benchmarks.
Project page: https://github.com/BT-C/AID