LogicPro: Logical Reasoning Enhanced with Program Examples
Abstract: In this paper, we present a novel approach, called \textbf{LogicPro}, to enhance \underline{Logic} reasoning through \underline{Pro}gram examples to improve multiple complex reasoning tasks simultaneously. We do this effectively by simply utilizing widely available algorithmic problems and their code solutions.
First, we constructed diverse input test samples based on algorithmic questions and code solutions. Then, we designed different logic reasoning questions based on the algorithmic problems and test samples. Finally, combining the intermediate variable outputs of the code solutions and the logic reasoning questions, we obtain the final reasoning path through a large language model.
Based on this, we are able to construct very rich \textit{SFT} data. At the same time, we construct a diverse and scalable dataset of logical reasoning evaluation by treating each algorithmic question as a reasoning rule.
As a result, our approach achieves significant improvements on multiple models for BBH dataset (20+ subsets), GSM8K and HellSwag datasets, and significantly outperforms a wide range of existing logical reasoning datasets. In addition, our eval data distinguishes well between existing models and brings new challenges to the model.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: logical reasoning,
Contribution Types: Data resources
Languages Studied: English
Submission Number: 5706
Loading