Attention Contrastive Decoding: Preserving Coherence While Mitigating Hallucinations in Large Vision-Language Models

ICLR 2026 Conference Submission25173 Authors

Keywords: Trustworthy AI, Hallucination Alleviation, Large Vision-Language Models

TL;DR: We propose an adaptive contrastive decoding approach at the attention layer to mitigate hallucinations and improve coherence in large vision-language models.

Abstract: Large Vision-Language Models (LVLMs) exhibit remarkable multimodal capabilities but frequently produce factually inconsistent hallucinations. While Contrastive Decoding (CD) methods offer a training-free approach to hallucination mitigation, they operate at the logits level, compromising output coherence and diversity. Through systematic analysis, we show that logits-level subtraction disrupts intrinsic language generation mechanisms, requiring restrictive penalty mechanisms that further limit diversity. We propose Attention Contrastive Decoding (ACD), which transfers contrastive operations to the attention layer and employs an Adaptive Subtraction Strategy (ASS) to identify and suppress hallucination-prone attention patterns. Experiments demonstrate that ACD generates more coherent content with significantly reduced hallucinations without requiring penalty mechanisms, effectively leveraging the inherent continuity of attention mechanisms to advance reliable multimodal generation. Code is available at \url{https://anonymous.4open.science/r/ACD-00C6}.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 25173