A New Perspective on Factual Knowledge Extraction in Large Language Models: Combining Fine-Tuning and Inference

ACL ARR 2025 February Submission5967 Authors

Abstract: Factual knowledge extraction aims to explicitly extract knowledge parameterized in pre-trained language models for application in downstream tasks. Recent work has been investigating the impact of fine-tuning on the factuality of large language models (LLMs). In this paper, we thoroughly study this impact through systematic experiments, with a particular focus on the factuality gap caused by unknown and known knowledge. We find that this gap is essentially a discrepancy between attention patterns, which can be influenced by both fine-tuning and in-context learning (i.e., few-shot learning and Chain of Thought (CoT)). Appropriate prompt design during the inference stage can even mitigate the factuality gap caused by fine-tuning. Therefore, we argue that both stages play essential roles in factual knowledge extraction, and that they need to be studied in combination. Finally, we seek to provide explanations and offer novel insights into factual knowledge extraction through the integration of fine-tuning and inference in LLMs.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: data influence, explanation faithfulness, knowledge inducing

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 5967