LeanReasoner: Boosting Complex Logical Reasoning with Lean

Anonymous

TL;DR: We address natural language logical reasoning problems with a Lean based theorem proving framework that makes use of the logic nuggets in mathematical theorem proving data

Abstract: Large language models (LLMs) frequently face challenges with complex logical reasoning tasks. We address this issue with the help of Lean, a theorem proving framework. First, we formalize logical reasoning problems as theorems within Lean and then proceed to either prove or disprove them. This methodology serves dual purposes: it eliminates the possibility of logical inconsistencies typical in LLM outputs and effectively manages complex logical reasoning tasks. Central to our approach are the numerous theorem proofs written in Lean, which encapsulate human logical reasoning. Training a model with this data enhances its capability to address logical reasoning problems. Our method demonstrates state-of-the-art performance on FOLIO dataset and achieves performance near this level on ProofWriter. Notably, these results were accomplished by fine-tuning on fewer than 100 in-domain samples for each dataset.

Paper Type: long

Research Area: Question Answering

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Data resources

Languages Studied: English