Revisit What You See: Disclose Language Prior in Vision Tokens for LVLM Decoding

Beomsik Cho, Jaehyung Kim

Keywords: LVLM, hallucination, vision tokens

Abstract:

Large Vision–Language Models (LVLMs) achieve strong performance across multimodal tasks by integrating visual perception with language understanding.
However, how vision information contributes to the model’s decoding process remains under-explored, as reflected in frequent hallucinations.
Through a series of analyses, we found that (i) vision tokens provide meaningful visual information even when when hallucinations occur, and (ii) their semantics are encoded in the textual space and become explicit under appropriate vocabulary constraints.
Building on these observations, we propose ReVisiT, a simple training-free decoding method that references vision tokens to guide text generation.
Our approach leverages the semantic information embedded within vision tokens by projecting them into the text token distribution.
Specifically, ReVisiT dynamically selects the most relevant vision token at each decoding step via context-aware constrained divergence minimization, and using its constrained projection to refine the output distribution to better incorporate visual semantics.
Across five benchmarks on recent LVLMs, ReVisiT consistently enhances visual grounding with minimal computational overhead, and achieves competitive or superior results to state-of-the-art decoding baselines while reducing computational cost by up to $2\times$.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 23483