Reducing Hallucinations in Multimodal Large Language Models via Causal Fusion

ICLR 2026 Conference Submission15841 Authors

Keywords: Large Vision-Language Models, Hallucination Mitigation, Causality

TL;DR: Causal decoding curbs object hallucination in MLLMs, improving faithfulness without quality loss.

Abstract: Multimodal Large Language Models (MLLMs) deliver detailed responses on vision-language tasks, yet remain susceptible to object hallucination (introducing objects not present in the image), undermining reliability in practice. Prior efforts often rely on heuristic penalties, post-hoc correction, or generic decoding tweaks, which do not directly intervene in the mechanisms that trigger object hallucination and thus yield limited gains. To address this challenge, we propose a causal decoding framework that applies targeted causal interventions during generation to curb spurious object mentions. By reshaping the decoding dynamics to attenuate spurious dependencies, our approach reduces false object tokens while maintaining descriptive quality. Across captioning and QA benchmarks, our framework substantially lowers object-hallucination rates and achieves state-of-the-art faithfulness without degrading overall output quality.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 15841