Judgment-of-Thought Prompting: A Courtroom-Inspired Framework for Binary Logical Reasoning with Large Language Models

ACL ARR 2025 May Submission2682 Authors

Abstract:

This paper proposes a novel prompting approach, Judgment of Thought (JoT), specifically tailored for binary logical reasoning tasks.
Despite advances in prompt engineering, existing approaches still face limitations in handling complex logical reasoning tasks.
To address these issues, JoT introduces a multi-agent approach with three specialized roles---lawyer, prosecutor, and judge---where a high-level model acts as the judge, and lower-level models serve as lawyer and prosecutor to systematically debate and evaluate arguments.
Experimental evaluations on benchmarks such as BigBenchHard and Winogrande demonstrate JoT's superior performance compared to existing prompting approaches, achieving notable improvements, including 98% accuracy in Boolean expressions.
Also, our ablation studies validate the critical contribution of each role, iterative refinement loops, and feedback mechanisms.
Consequently, JoT significantly enhances accuracy, reliability, and consistency in binary reasoning tasks and shows potential for practical applications.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: prompting, LLM/AI Agents, chain-of-thought

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 2682