ProRAC: A Neuro-symbolic Method for Reasoning about Actions with LLM-based Progression

Haoyong Wu, Yongmei Liu

Keywords: Large Language Model, Reasoning, Reasoning about Actions and Change, Prompting, Neuro-Symbolic Method

TL;DR: We propose ProRAC, a neuro-symbolic framework targeting RAC problems effectively.

Abstract: Reasoning about actions and change (RAC) plays an important role in AI. It involves reasoning about preconditions and effects of actions, and has applications in planning, an important research area in classic AI. In recent years, large language models (LLMs) have made remarkable progress in NLP and related fields, and reasoning over natural language has received much attention. Several benchmarks for RAC over natural language such as TRAC, ActionReasoningBench and ACPBench have been proposed. Evaluation of LLMs on these benchmarks demonstrates that LLMs face significant challenges in RAC. However, methods for improving RAC abilities of LLMs remain largely unexplored. In this paper, we propose ProRAC (Progression-based Reasoning about Actions and Change), a neuro‐symbolic framework that leverages LLMs to tackle RAC problems. ProRAC extracts fundamental RAC elements including actions and questions from the problem, progressively executes each action to derive the final state, and then evaluates the query against the progressed state to arrive at an answer. We evaluate ProRAC on several RAC benchmarks, and the results demonstrate that our approach achieves strong performance across different benchmarks, domains, LLM backbones, and types of RAC tasks.

Submission Number: 51