BALCONI: BALancing CONtext and Internal Knowledge For Training Flexible LLMs

Yun Luo, Xiangkun Hu, Qinglin Qi, Qipeng Guo, Zheng Zhang, Yue Zhang

Keywords: Large Language Model, Faithfulness, Internal knowledge

Abstract: The faithfulness to the context is significant for large language models (LLMs) in tasks such as Retrieval-Augmented Generation (RAG) or Information Extraction. However, LLMs can exhibit a "stubborn" reliance on their internal knowledge, which leads to failure in maintaining faithfulness to the context. Ideally, a model should leverage the given context if the user instruction requires to, yet remain correctness based on internal knowledge when the instruction does not provide the context. Considering such scenarios, we propose a balanced benchmark, FaithfulBench, to evaluate the faithfulness of LLMs, together with internal knowledge correctness in LLMs and evaluate whether the improvement in faithfulness would affect internal knowledge. Extensive experiments show that LLMs can be unfaithful to the context to some extent and in the Multi-choice QA, we observe an obvious negative correlation between faithfulness and internal knowledge correctness across different LLMs. Then based on the analysis of faithfulness enhancement methods, we find that instruction tuning using counterfactual data can significantly improve the model's context faithfulness, but compromise the model's internal knowledge. To address such a issue, we propose a straightforward yet effective approach BALCONI training by training with mixup data of factual requests, context requests, and NoAns (I cannot tell the answer from the context) requests. Experiments on our benchmark and a context-based machine translation task demonstrate that BALCONI can achieve a well-balanced effect in improving the balanced faithfulness and internal knowledge.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6787