Retrieval over Large Language Model's Latent Causal Knowledge Graph for Deductive Reasoning

Anonymous

Abstract: Deductive reasoning refers to the task of drawing conclusions based on a premise. While some deductive reasoning benchmarks exist, none focus on causal deductive reasoning and are from real-world applications. Therefore, this paper explores the causal deductive reasoning task conducted by Accident Investigators, who analyze accidents to determine probable causes. Recently, large language models (LLMs) used with prompt engineering techniques like retrieval-augmented generation (RAG) have demonstrated remarkable performance across various natural language processing benchmarks. However, adapting these techniques to handle scenarios with no knowledge bases and to different data structures, such as graphs, remains an ongoing challenge. In our study, we introduce a novel framework leveraging LLMs' decent ability to detect and infer causal relations to construct a causal Knowledge Graph (KG) which represents knowledge that the LLM recognizes. Additionally, we propose a RoBERTa-based Transformer Graph Neural Network (RoTG) specifically designed to select relevant nodes within this KG. Integrating RoTG-retrieved causal chains into prompts effectively enhances LLM performance, demonstrating usefulness of our approach in advancing LLMs' causal deductive reasoning capabilities.

Paper Type: long

Research Area: NLP Applications

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: English

Preprint Status: There is no non-anonymous preprint and we do not intend to release one.

A1: yes

A1 Elaboration For Yes Or No: Section 8

A2: yes

A2 Elaboration For Yes Or No: Section 8

A3: yes

A3 Elaboration For Yes Or No: See Abstract and Introduction

B: yes

B1: yes

B1 Elaboration For Yes Or No: Yes, Section 3 and 8

B2: yes

B2 Elaboration For Yes Or No: Will be mentioned in the Github repository

B3: yes

B3 Elaboration For Yes Or No: Yes, Section 3 describes how we processed the dataset

B4: yes

B4 Elaboration For Yes Or No: Yes, the dataset is already originally anonymized.

B5: yes

B5 Elaboration For Yes Or No: Section 3

B6: yes

B6 Elaboration For Yes Or No: Section 3

C: yes

C1: yes

C1 Elaboration For Yes Or No: Appendix Section A

C2: yes

C2 Elaboration For Yes Or No: Appendix Section A

C3: yes

C3 Elaboration For Yes Or No: Throughout paper, all findings are reported with statistical significance across 10-folds.

C4: yes

C4 Elaboration For Yes Or No: Section 3.1 for evaluation

D: no

D1: n/a

D2: n/a

D3: n/a

D4: n/a

D5: n/a

E: yes

E1: yes

E1 Elaboration For Yes Or No: We use Claude 2.1, Mistral-Instruct. Described in Section 3 and 5