Reasoning Court: Combining Reasoning, Action, and Judgment for Multi-Hop Reasoning

ACL ARR 2024 December Submission750 Authors

Abstract: While large language models (LLMs) have significantly advanced tasks such as question answering and fact verification, they continue to grapple with hallucinations and reasoning errors, especially in multi-hop tasks that require integrating information from multiple sources. Current research primarily follows two approaches: (1) retrieval-based methods, which ground reasoning in external data to mitigate hallucinations, and (2) reasoning-based techniques, which enhance logical consistency through improved prompting strategies. In this paper, we introduce Reasoning Court (RC), a novel framework where LLM agents iteratively reason and act, generating distinct reasoning-action-observation trajectories. These trajectories are then evaluated by a judge, who selects the most factually grounded and logically coherent final answer based on the reasoning paths. If neither answer is satisfactory, the judge synthesizes a new answer using the evidence and reasoning provided by both agents. This process ensures that the final response is both evidence-based and logically consistent, significantly reducing reasoning flaws. Our evaluations on HotpotQA, MuSiQue, and FEVER demonstrate that RC consistently outperforms state-of-the-art approaches.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: multihop QA, logical reasoning, reasoning

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data analysis

Languages Studied: English

Submission Number: 750