Logical forms complement probability in understanding language model (and human) performance

ACL ARR 2025 February Submission2674 Authors

Abstract:

With the increasing interest in using large language models (LLMs) for planning in natural language, understanding their behaviors becomes an important research question.
This work conducts a systematic investigation of LLMs' ability to perform logical reasoning in natural language.
We introduce a controlled dataset of hypothetical and disjunctive syllogisms in propositional and modal logic and use it as the testbed for understanding LLM performance.
Our results lead to novel insights in predicting LLM behaviors: in addition to the probability of input, logical forms should be considered as important factors.
In addition, we show similarities and discrepancies between the logical reasoning performances of humans and LLMs by collecting and comparing behavioral data from both.

Paper Type: Long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics

Research Area Keywords: cognitive modeling, computational psycholinguistics

Contribution Types: Model analysis & interpretability, Data resources, Surveys

Languages Studied: English

Submission Number: 2674