Why Does ChatGPT Fall Short in Answering Questions Faithfully?

Shen Zheng, Jie Huang, Kevin Chen-Chuan Chang

Abstract: Recent advancements in Large Language Models, such as ChatGPT, have demonstrated significant potential to impact various aspects of human life. However, ChatGPT still faces challenges in aspects like truthfulness, e.g. providing accurate and reliable outputs. Therefore, in this paper, we seek to understand why ChatGPT falls short in providing truthful answers. For this purpose, we first analyze the failures of ChatGPT in complex open-domain question answering and identifies the abilities under the failures. Specifically, we categorize ChatGPT's failures into four types: comprehension, factualness, specificity, and inference. We further pinpoint three critical abilities associated with QA failures: knowledge memorization, knowledge recall, and knowledge reasoning. Additionally, we conduct experiments centered on these abilities and propose potential approaches to enhance truthfulness. The results indicate that furnishing the model with fine-grained external knowledge, hints for knowledge recall, and guidance for reasoning can empower the model to answer questions more truthfully.