Reasoning Abilities of Large Language Models through the Lens of Abstraction and Reasoning

Seungpil Lee, Woochang Sim, Donghyeon Shin, Sejin Kim, Sundong Kim

Keywords: ARC, LLM, Reasoning, Language of thought hypothesis

TL;DR: LLMs struggle with consistent rule application, combining simple operations for complex tasks, and generating valid new examples. These findings highlight the gap between current LLM capabilities and human-like reasoning in abstract problem-solving.

Abstract: Large Language Models (LLMs) have recently demonstrated impressive capabilities across a range of natural language processing tasks. However, a fundamental question remains: to what extent do these models exhibit genuine reasoning abilities? In this study, we focus on understanding the inference processes of LLMs through an in-depth evaluation of their reasoning capabilities on tasks drawn from the Abstraction and Reasoning Corpus (ARC). Our approach takes inspiration from the "Language of Thought" Hypothesis (LoTH), which posits that human reasoning is built upon three core components: logical coherence, compositionality, and productivity. By evaluating LLMs on these three dimensions, we aim to provide insights into their reasoning strengths and limitations. Through this extended abstract, we highlight key experimental results that illuminate the capabilities and limitations of current LLMs in tasks requiring advanced cognitive reasoning.

Submission Number: 78