Context Over Memory: Self-Supervised Faithfulness Optimization via Likelihood Displacement

ACL ARR 2026 January Submission792 Authors

Keywords: Faithfulness, Preference Optimization, Likelihood Displacement

Abstract:

Mitigating context-faithfulness hallucinations is crucial for grounding large language models (LLMs) in the provided context.
However, most existing approaches require costly supervision and post-training, or impose significant inference burdens.
To overcome these limitations, we introduce Self-Supervised Faithfulness Optimization (SSFO), a self-supervised alignment approach for enhancing faithfulness. SSFO constructs preference data pairs by contrasting the model's outputs generated with context versus without context. Leveraging Direct Preference Optimization (DPO), SSFO aligns model faithfulness without incurring labeling costs or additional inference burdens.
We analyze this faithfulness alignment process and provide empirical evidence that it leverages a benign form of likelihood displacement, shifting probability mass from parametric-based tokens to context-aligned tokens.
Based on this insight, we adapt the DPO loss using a weighting scheme that encourages likelihood displacement. Comprehensive evaluations show that SSFO significantly outperforms existing methods, achieving state-of-the-art results in faithfulness on multiple context-based question-answering datasets. Notably, SSFO exhibits strong generalization, improving cross-lingual faithfulness while preserving general instruction-following capabilities. The code is available at: https://anonymous.4open.science/r/SSFO

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: Misinformation detection, NLP Applications

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Publicly available software and/or pre-trained models

Languages Studied: English, Spanish, Chinese

Submission Number: 792