Large Language Models Exhibit Limited Reasoning Ability on Coding Problems

ACL ARR 2025 February Submission2956 Authors

Abstract:

Claims that large language models (LLMs) have complex reasoning ability have stirred broad interests, and controversies, of academics and non-academics alike.
A popular basis for such claims comes from LLMs' ability to solve coding problems, which involves understanding the problem statement and providing code that solves the problem.
Although such abilities are remarkable feats worth praising, we argue that they come from memorization rather than reasoning.
We first show that LLMs' problem-solving ability degrades with increased recency of the problem, likely due to the reduced amount of training data for more recent problems, regardless of the problem difficulty labeled by human experts.
Additionally, we show that an LLM often fails to solve the problem when presented with reworded but equivalent problem statements, further suggesting their limited reasoning ability.

Paper Type: Short

Research Area: Question Answering

Research Area Keywords: logical reasoning, reasoning, generalization

Contribution Types: Model analysis & interpretability, Position papers

Languages Studied: English

Submission Number: 2956