Enhancing Knowledge through Revisable Chain-of-Thought for Commonsense Question Answering

ACL ARR 2024 August Submission144 Authors

Abstract: Large Language Models (LLMs) are effective at natural language reasoning, but still struggle with answering commonsense questions that require implicit knowledge of the world. LLMs rely on knowledge learned through training, which can be limited to specific domains and may lack inductive abstraction, resulting in hallucinations and inaccurate knowledge. To alleviate these, recent research integrates external knowledge sources (e.g., fine-tuning, self-correction, retrieval enhancement, and chain-of-thought (CoT)). While CoT reveals specific incorrect knowledge in LLMs, it lacks abstraction and is uneasy to be revised. In this paper, we propose a revisable three-step CoT framework, categorizing knowledge into abstract meta-knowledge and concrete instantiated knowledge. Meanwhile, we use transfer knowledge to address the logical form sensitivity of LLMs. Furthermore, we propose online revision by teacher models and offline revision with knowledge base. We propose an antisense retrieval method to check if the newly generated knowledge contradicts any existing knowledge in the knowledge base to avoid retrieving meta-knowledge that is not relevant to the problem. The experimental results on the Winogrande dataset have corroborated the efficacy of our proposed method. We revised the meta-knowledge of GPT-3.5 with GPT-4, which enhanced the accuracy from 68.11% to 73.64%, an improvement of 5.53 percentage points.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: Question Answering,Human-Centered NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 144