FreqAlign: Frequency-Based Calibration for Mitigating Contextual Hallucinations in Large Language Models

Zeyu Gao, Yizhang Liu

Keywords: Large language models (LLMs)​​, Context hallucinations​, ​ Frequency alignment​, ​Context-aware classifier

Abstract: Despite significant progress, large language models (LLMs) continue to exhibit context hallucinations, generating content that either contradicts or fabricates information relative to the retrieved context. To address this issue, we introduce FreqAlign, a frequency alignment-based method designed to mitigate such hallucinations. Our approach consists of two main stages: first, we construct positive and negative sample pairs according to the actual contextual influence on tokens and train a context-aware classifier that evaluates contextual relevance using token frequency information; second, this classifier is employed to recalibrate the importance distribution of original tokens via frequency alignment. A core innovation of FreqAlign is its dual strategy: it suppresses globally frequent tokens that are prone to induce hallucinations, while enhancing semantically salient yet contextually infrequent tokens. We evaluate our method across six widely-used question-answering benchmarks, where it consistently and substantially outperforms strong state-of-the-art baselines. The implementation of FreqAlign, including training scripts, evaluation protocols, and hyperparameter configurations, is publicly available at https://anonymous.4open.science/r/FreqAlign-BC8E . This open release enables reproducibility and facilitates future research into frequency-aware prompting and decoding strategies for reducing hallucinations in LLMs. These findings underscore the potential of frequency-aware modeling as a general and effective strategy for reducing hallucinations in LLMs.

Primary Area: optimization

Submission Number: 10844