Solver-Guided Optimization of Large Language Models for Logic Puzzle Reasoning with Answer Set Programming

ACL ARR 2025 May Submission4155 Authors

Abstract:

The rise of large language models (LLMs) has sparked interest in neuro-symbolic systems that leverage logic reasoners to overcome LLM shortcomings. Answer set programming (ASP) is a particularly effective approach to finding solutions to combinatorial search problems, which LLMs often fail to solve. However, the effectiveness of LLMs in ASP
code generation is hindered by the limited number of examples seen during their initial pre-training phase.

In this paper, we introduce a novel approach for solver-guided instruction-tuning of LLMs for addressing the highly complex semantic parsing task inherent in ASP code generation. We sample ASP statements for program continuations proposed by LLMs for unriddling logic puzzles and categorize them into chosen and rejected instances based on solver feedback. We then apply supervised fine-tuning to train LLMs on the curated data, and further improve robustness using best-of-N test-time sampling. Our experiments demonstrate consistent improvements across four datasets.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: logical reasoning; semantic parsing; reasoning

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English

Submission Number: 4155