FLAME : Factuality-Aware Alignment for Large Language Models

Sheng-Chieh Lin, Luyu Gao, Barlas Oguz, Wenhan Xiong, Jimmy Lin, Wen-tau Yih, Xilun Chen

Keywords: large language models, factuality, alignment

TL;DR: We find that the standard alignment process encourages hallucination, and propose factuality-aware alignment while maintaining the LLM's general instruction-following capability.

Abstract:

Alignment is a procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants.
We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e., *hallucination*).
In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps: supervised fine-tuning (SFT) and reinforcement learning (RL).
In particular, we find that training the LLM on new or unfamiliar knowledge can encourage hallucination.
This makes SFT less factual as it trains on human-labeled data that may be novel to the LLM.
Furthermore, reward functions used in standard RL often inadequately capture factuality and favor longer and more detailed responses, which inadvertently promote hallucination.
Based on these observations, we propose *FactuaLity-aware AlignMEnt*, comprised of *factuality-aware SFT* and *factuality-aware RL* through direct preference optimization.
Experiments show that our proposed *FLAME* guides LLMs to output more factual responses while maintaining their instruction-following capability.

Primary Area: Natural language processing

Submission Number: 1553