MTRE: Multi-Token Reliability Estimation for Hallucination Detection in VLMs

ICLR 2026 Conference Submission13286 Authors

Keywords: alignment, fairness, safety, privacy

TL;DR: MTRE method improves reliability detection by aggregating multi-token logits, outperforming prior approaches efficiently despite large vocabularies and logit sequences.

Abstract:

Vision–language models (VLMs) now rival human performance on many multimodal tasks, yet they still hallucinate objects or generate unsafe text. Current hallucination detectors, e.g., single-token linear probing (LP) and $P(\text{True})$, typically analyze only the logit of the first generated token—or just its highest-scoring component—overlooking richer signals embedded within earlier token distributions. We demonstrate that analyzing the complete sequence of early logits potentially provides substantially more diagnostic information.
We emphasize that hallucinations may only emerge after several tokens, as subtle inconsistencies accumulate over time. By analyzing the Kullback–Leibler (KL) divergence between logits corresponding to hallucinated and non-hallucinated tokens, we underscore the importance of incorporating later-token logits to more accurately capture the reliability dynamics of VLMs.
In response, we introduce Multi-Token Reliability Estimation (MTRE), a lightweight, white-box method that aggregates logits from the first ten tokens using multi-token log-likelihood ratios and self-attention. Despite the challenges posed by large vocabulary sizes and long logit sequences, MTRE remains efficient and tractable. Across MAD-Bench, MM-SafetyBench, MathVista, and four compositional-geometry benchmarks, MTRE achieves a 9.4% gain in Accuracy and a 14.8% gain in AUROC over standard detection methods, establishing a new state of the art in hallucination detection for open-source VLMs.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 13286