A Survey on Large Language Model Reasoning Failures

Peiyang Song, Pengrui Han, Noah Goodman

Keywords: Large Language Models, Failure Cases, Cognitive Reasoning, Logical Reasoning, Embodied Reasoning

TL;DR: We present the first comprehensive survey that unifies the previously overlooked, important field of LLM reasoning failures, and provides insights for future research.

Abstract:

Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, achieving impressive results across a wide range of tasks.
Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios.
To systematically understand and address these shortcomings, we present the first comprehensive survey dedicated to reasoning failures in LLMs.
We introduce a novel categorization framework that distinguishes reasoning into embodied and non-embodied types, with the latter further subdivided into informal (intuitive) and formal (logical) reasoning.
In parallel, we classify reasoning failures along a complementary axis into three types: fundamental failures intrinsic to LLM architectures that broadly affect downstream tasks; application-specific limitations that manifest in particular domains; and robustness issues characterized by inconsistent performance across minor variations.
For each category, we synthesize existing studies, analyze common failure patterns and underlying causes, and suggest mitigation strategies.
By unifying fragmented research efforts, our survey provides a structured perspective on systemic weaknesses in LLM reasoning, offering valuable insights and guiding future research towards building stronger, more reliable, and robust reasoning capabilities.

Submission Number: 8