CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMs

Kangsheng Wang, Xiao Zhang, Huimin Ma, Tianyu Hu, Hao Liu, Songde Han

Keywords: Causal Reasoning Enhancement, Dual-End Searching, Long-Range Reasoning, LLM

TL;DR: This paper presents an LLM-based method using Causal Reasoning Enhancement and Dual-End Searching to address long-range reasoning problems, guaranteeing both accuracy and time efficiency.

Abstract: Large language models (LLMs) have demonstrated limitations in handling combinatorial optimization problems involving long-range reasoning, partially due to causal hallucinations and huge search space. As for causal hallucinations, i.e., the inconsistency between reasoning and corresponding state transition, this paper introduces the Causal Relationship Enhancement (CRE) mechanism combining cause-effect interventions and the Average Treatment Effect (ATE) to guarantee the solid causal rightness between each step of reasoning and state transition. As for the long causal range and huge search space limiting the performances of existing models featuring single-direction search, a Dual-End Searching (DES) approach is proposed to seek solutions by simultaneously starting from both the initial and goal states on the causal probability tree. By integrating CRE and DES (CreDes), our model has realized simultaneous multi-step reasoning, circumventing the inefficiencies from cascading multiple one-step reasoning like the Chain-of-Thought (CoT). Experiments demonstrate that CreDes significantly outperforms existing State-Of-The-Art (SOTA) solutions in long-range reasoning tasks in terms of both accuracy and time efficiency.

Primary Area: Causal inference

Submission Number: 6100