DOUBT: Decoupled Object-level Understanding and Bridging via vMF-based Trustworthiness for Hallucination Detection in MLLMs

Kaiqi Chen, Yang Qin, Changhao He, Xi Peng, Peng Hu

TL;DR: This paper proposes a hallucination detection method that incorporates an object-level bridging mechanism based on MLLMs and leverages the mean resultant length to measure uncerntianty inspired by the von Mises–Fisher distribution.

Abstract: Multimodal Large Language Models (MLLMs) frequently produce hallucinations (i.e., assertions that contradict the image or facts), undermining reliability in high-risk applications. Existing detection approaches typically feed images and texts jointly and estimate hallucination scores by measuring the consistency of model outputs. However, because the visual module often lags behind the language module in understanding and reasoning, MLLMs can repeatedly produce similar yet incorrect answers, yielding overestimated trustworthiness and missed detections. To address this, we propose a simple yet effective model-agnostic method, dubbed Decoupled Object-level Understanding and Bridging via vMF-based Trustworthiness (DOUBT). DOUBT first employs Object-level Understanding and Bridging (OUB), a two-step prompting scheme that decouples object recognition from relational reasoning by prompting the model to identify objects and then reason based on them. It further introduces a von Mises-Fisher (vMF)-based trustworthiness metric, which is more stable than semantic entropy metrics in small-sample settings. Extensive experiments and ablation studies on multiple benchmarks show that DOUBT consistently outperforms state-of-the-art baselines, demonstrating its robustness and generalizability for hallucination detection in MLLMs. The code is available at https://github.com/XLearning-SCU/2026-ICML-DOUBT.

Lay Summary:

When advanced AI models look at images to answer questions, they often confidently make up false facts, a dangerous flaw called "hallucination." Traditional guardrails try to catch these errors by checking if the AI gives consistent answers over multiple attempts. However, because an AI's language reasoning often outpaces its visual understanding, it frequently repeats the exact same mistake with stubborn consistency, completely tricking existing detectors into validating incorrect answers.

To break this cycle of blind confidence, we introduce DOUBT, a method that forces the AI to first explicitly list the core visual objects it sees before reasoning out its final response. This two-step strategy shatters false consistency by exposing hidden contradictions when the AI is guessing or hallucinating. DOUBT then evaluates these generated responses using geometric similarity calculations to track how closely they align; if the AI is fabricating information, the answers mathematically scatter, providing a highly stable truth check even with very few samples.

Ultimately, this tool significantly boosts hallucination detection accuracy across multiple standard benchmarks and model scales. Because it functions as a plug-and-play solution running purely on clever questioning and math, it requires no core model modifications or expensive external databases, making real-world AI deployment much safer.

Link To Code: https://github.com/XLearning-SCU/2026-ICML-DOUBT

Primary Area: Deep Learning

Keywords: Hallucination Detection; Object-Level Understanding and Bridging; von Mises–Fisher Distribution

Submission Number: 24632