Information-Theoretic Bounds on Multi-Step Reasoning: When is Chain-of-Thought Provably Necessary?

Karthik Srikumar

Keywords: Large Reasoning Models, Chain-of-Thought, Sample Complexity, Information Bottlenecks, Chain-of-Thought Reasoning, Information Theory, Sample Complexity, Multi-Step Reasoning, PAC Learning

TL;DR: We establish when intermediate reasoning steps are provably necessary, providing exponential sample complexity separations

Abstract:

Chain-of-Thought (CoT) reasoning has emerged as a powerful technique for enhancing the performance of large reasoning models on complex tasks. However, a fundamental question remains unresolved: when is intermediate reasoning provably necessary, and when can direct prediction suffice?

We establish the first information-theoretic framework for characterizing the necessity of multi-step reasoning. Our main result proves that for a broad class of compositional reasoning tasks, any model that directly maps inputs to outputs without intermediate steps must suffer an exponential degradation in sample complexity compared to models that perform explicit chain-of-thought reasoning.

Specifically, we show that the mutual information between inputs and outputs, when mediated through a sequence of reasoning steps, can be exponentially larger than the direct mutual information, a phenomenon we formalize as reasoning information gain. We further characterize problem classes where CoT provides polynomial versus exponential advantages, establishing a complexity hierarchy based on reasoning depth, problem compositionality, and information bottlenecks.

Submission Number: 238