AdaptiveResidual: Inference-Time Trust Calibration for Contextual Knowledge Injection

ICLR 2026 Conference Submission771 Authors

Keywords: Knowledge Conflict, In-Context Knowledge Injection

Abstract:

In modern large language models (LLMs), injecting external knowledge via the context to guide models' outputs toward desired outcomes (e.g., through RAG) is a standard practice.
However, recent research reveals that once conflicts arise between the contextual information and the internal parametric knowledge, LLMs tend to underutilize the external evidence, leading to unreliable or even contradictory outputs.
This raises a fundamental question: *how can we dynamically reconcile these knowledge conflicts to ensure faithful integration of contextual information* ?
Inspired by mechanism interpretability findings that identify the `Attention` module as the primary aggregator of external context and the `FFN` module as the locus of internal knowledge lookup, we pinpoint the vanilla residual pathway as the crucial junction where these two information streams are integrated.
Based on this insight, we introduce AdaRes (*Ada*ptive *Res*idual), a lightweight, parameter-free trust calibration mechanism that operates at test-time.
Specifically, AdaRes recalibrates the standard residual connection to dynamically balance the influence of external knowledge (from `Attention`) and internal knowledge (from `FFN`).
This balancing is guided by two instance-specific "trust scores", which are calculated on-the-fly by probing how much the input query relies on contextual versus parametric knowledge sources.
By adaptively reweighting these contributions without altering any model parameters, AdaRes effectively mitigates knowledge conflicts.
Experiments on different benchmarks verify the effectiveness of AdaRes in regulating contextual and parametric knowledge.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 771