FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness

Chengyan Wu, Xiaoning Dong, Yajie Wen, Yu Chen, Yun Xue, Zhang Jing, Wei Xu, Bolei Ma

Keywords: Factuality Alignment, Uncertainty Quantification, Reward Modeling, Knowledge Integration

TL;DR: We propose FAITH, which improves LLM factuality by first teaching it to self-assess its knowledge in natural language. It then guides a PPO-based alignment process, which is followed by a retrieval-augmented module for external grounding.

Abstract: Large Language Models (LLMs) can generate factually inaccurate content even if they have corresponding knowledge, which critically undermines their reliability. Existing approaches attempt to mitigate this by incorporating uncertainty in QA prompt during training, but these numerical scores lack the semantic richness for LLM to properly understand its internal states of trustworthiness and honestness, leading to insufficient factuality alignment. We introduce FAITH (Factuality Alignment through Integrating Trustworthiness and Honestness), a post-training framework for factuality alignment that integrates natural-language uncertainty signals with external knowledge. Specifically, we augment training datasets by computing confidence scores and semantic entropy from LLM outputs and mapping them into a knowledge state quadrant that describes the model's internal knowledge possession (trustworthiness) and answering behaviors (honestness) in natural language. Based on this enhanced data, we design a reward function that considers both correctness and uncertainty signals, and fine-tune the LLM using the Proximal Policy Optimization (PPO) algorithm. To further mitigate weakly grounded responses, we design a retrieval-augmented module that retrieves relevant external passages, improving the consistency between internal and external knowledge representations. Extensive experiments on four knowledge-intensive benchmarks demonstrate that FAITH enhances the factual accuracy and truthfulness of LLMs.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 7183