Can Language Models Pretend Solvers? Logic Code Simulation with LLMs

Minyu Chen, Guoqiang Li, Ling-I Wu, Ruibang Liu, Yuxin Su, Xi Chang, Jianxin Xue

Abstract: Logical solvers are typically employed for the static analysis and formal verification of functional code that is not amenable to verification through execution. Concurrently, some research efforts aim to convert natural language into logic code to bolster reasoning capabilities with the aid of logical solvers. However, the transformation process from code or natural language to a logical framework is costly. To address this challenge, several studies have explored the use of large language models (LLMs) to facilitate this transformation. Nevertheless, LLMs may introduce misalignment and formatting errors that are unacceptable for logical solvers. Considering the strong logical reasoning capabilities of LLMs and their relative insensitivity to alignment and formatting errors, we propose utilizing LLMs to emulate logical solvers for logic code simulation. In light of the characteristic fixed outputs of logical solvers, we introduce a novel LLM-based code simulation technique called Dual Chains of Logic (DCoL). Additionally, we have compiled three new datasets specifically for the Logic Code Simulation task and conducted comprehensive evaluations. These assessments demonstrate state-of-the-art performance when compared to other LLM prompt strategies, achieving a notable 7.06% improvement in accuracy with GPT-4 Turbo.

External IDs:dblp:conf/setta/ChenLWLSCX24