Understanding Syllogistic Reasoning in LLMs from Formal and Natural Language Perspectives

Aheli Poddar, Saptarshi Sahoo, Sujata Ghosh

Keywords: Large Language Models, Syllogistic Reasoning, Belief Bias, Natural Language Understanding

TL;DR: We introduce a dual ground truth framework to evaluate syllogistic reasoning in 14 LLMs, finding they excel at formal logic while struggling with natural language plausibility, with better models showing less belief bias than humans.

Abstract: We study syllogistic reasoning in LLMs from the logical and natural language perspectives. In process, we explore fundamental reasoning capabilities of the LLMs and the direction this research is moving forward. To aid in our studies, we use 14 large language models and investigate their syllogistic reasoning capabilities in terms of symbolic inferences as well as natural language understanding. Even though this reasoning mechanism is not a uniform emergent property across LLMs, the perfect symbolic performances in certain models make us wonder whether LLMs are becoming more and more formal reasoning mechanisms, rather than making explicit the nuances of human reasoning.

Submission Number: 91