Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision-Language Models

ICLR 2026 Conference Submission10960 Authors

Keywords: Large Vision-Language Models, Multimodal Large Language Model, Hallucination

TL;DR: RUDDER is a low-overhead framework that reduces object hallucination in LVLMs by adaptively applying a corrective signal extracted during a single forward pass, matching the performance of computationally expensive state-of-the-art methods.

Abstract:

Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating text inconsistent with visual inputs, which can critically undermine their reliability. Existing inference-time interventions to mitigate this issue present a challenging trade-off: while methods that steer internal states or adjust output logits can be effective, they often incur substantial computational overhead, typically requiring extra forward passes. This efficiency bottleneck can limit their practicality for real-world, latency-sensitive deployments.
In this work, we aim to address this trade-off with **Residual-Update Directed DEcoding Regulation (RUDDER)**, a low-overhead framework that steers LVLMs towards visually-grounded generation. RUDDER is built on two key innovations: (1) Contextual Activation Residual Direction (CARD) vector, a per-sample visual evidence vector extracted from the residual update of a self-attention layer during a *single, standard forward pass*. (2) A Bayesian-inspired adaptive gate that performs token-wise injection, applying a corrective signal whose strength is conditioned on the model's deviation from the visual context.
Extensive experiments on key hallucination benchmarks, including POPE and CHAIR, indicate that RUDDER achieves performance comparable to state-of-the-art methods while introducing negligible computational latency, validating RUDDER as a pragmatic and effective approach for improving LVLMs' reliability without a significant compromise on efficiency.
Code is available at https://anonymous.4open.science/r/RrUuDdDdER-1C13/.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 10960