A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models

Tiwalayo Eisape, Michael Henry Tessler, Fei Sha, Ishita Dasgupta, Sjoerd van Steenkiste, Tal Linzen

Keywords: Language models, Logic, Reasoning, Syllogisms, Cognitive Science

TL;DR: We systematically evaluate language models on many syllogistic reasoning problems and show that while they are much more accurate than humans, they still show several humanlike biases and fallacies.

Abstract: A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises. Psychologists have documented several ways in which humans' inferences deviate from the rules of logic. Do language models, which are trained on text generated by humans, replicate these biases, or are they able to overcome them? Focusing on the case of syllogisms -inferences from two simple premises, which have been studied extensively in psychology - we show that larger models are more logical than smaller ones, and also more logical than humans. At the same time, even the largest models make systematic errors, some of which mirror human reasoning biases such as ordering effects and logical fallacies. Overall, we find that language models mimic the human biases included in their training data, but are able to overcome them in some cases.

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8303