Attention Smoothing: Correcting Causal Bias in Autoregressive Language Models

ICLR 2026 Conference Submission24845 Authors

Keywords: Large Language Models, Hallucination Mitigation, Attention Smoothing

Abstract: Autoregressive large language models (LLMs) suffer from causal bias: once attention states are cached under the causal mask, they cannot be revised, leading to information solidification and path-dependent errors. This structural limitation undermines contextual fidelity and amplifies hallucinations. We introduce Attention Smoothing, a decoding-time framework that revises attention after the entire context is observed. Our method models token-to-token information flow as an absorbing Markov chain, computes token-level surprisal scores, and derives a smoothed posterior attention distribution that corrects the causal bias. The framework is model-agnostic, training-free, and can be seamlessly integrated into existing inference pipelines. Experiments on multiple hallucination and factuality benchmarks show that Attention Smoothing consistently improves contextual faithfulness across model scales, highlighting the importance of managing information flow for more reliable LLM generation.

Primary Area: generative models

Submission Number: 24845