CoT-Space: A Theoretical Framework for Internal Slow-Thinking via Reinforcement Learning

Zeyu Gan, Hao Yi, Yong Liu

Keywords: large language model, reasoning, test-time scaling

TL;DR: This paper introduces CoT-Space, a novel theoretical framework that recasts LLM reasoning as a continuous optimization problem , which provides a coherent explanation for empirical phenomena such as overthinking.

Abstract: Reinforcement Learning (RL) has become a pivotal approach for enhancing the reasoning capabilities of Large Language Models (LLMs). However, a significant theoretical gap persists, as traditional token-level RL frameworks fail to align with the reasoning-level nature of complex, multi-step thought processes like Chain-of-Thought (CoT). To address this challenge, we introduce CoT-Space, a novel theoretical framework that recasts LLM reasoning from a discrete token-prediction task to an optimization process within a continuous, reasoning-level semantic space. This shift in perspective serves as a conceptual bridge, revitalizing foundational principles from classical learning theory to analyze the unique dynamics of LLMs. By analyzing this process from both a noise perspective and a risk perspective, we demonstrate that the convergence to an optimal CoT length is a natural consequence of the fundamental trade-off between underfitting and overfitting. Furthermore, extensive experiments provide strong empirical validation for our theoretical findings. Our framework not only provides a coherent explanation for empirical phenomena such as overthinking but also offers a solid theoretical foundation to guide the future development of more effective and generalizable reasoning agents. We open-source our code through an anonymous GitHub repository at https://anonymous.4open.science/r/CoT-Space-Reasoning-via-RL.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 9284