Dissecting Implicit Chain of Thought: Can Transformers Learn It Spontaneously?

ICLR 2026 Conference Submission16860 Authors

Keywords: transformers, LLM, Implicit Chain of Thought, Latent Chain of Thought

TL;DR: This paper primarily explains the reasons behind the difficulty of efficiently learning reasoning abilities in current implicit reasoning chains and proposes a distribution alignment method to guide the learning of implicit tokens.

Abstract: Large language models have gained powerful reasoning abilities through step-by-step reasoning chains, enabling deep logical reasoning and complex task handling. However, the shift from fast thinking to slow thinking (i.e., lengthy explicit reasoning steps) leads to massive token consumption, severely compromising practical reasoning efficiency. To address this, existing studies attempt to compress reasoning chains into latent tokens (Implicit CoT), but they often suffer from significant performance loss due to inadequate expression of original reasoning semantics. This paper theoretically analyzes the additional training costs of Implicit CoT when latent tokens lack effective supervision: as more steps are compressed, the originally efficient learning chain learning degrades exponentially, even reverting to the performance of "no CoT" when all intermediate processes are compressed. To solve this, we propose a distribution alignment method that adds moderate supervisory information to guide latent token distribution. Experimental results show that intermediate state supervision effectively improves the learning efficiency and stability of implicit CoT, significantly mitigating its reasoning performance decline.

Primary Area: learning theory

Submission Number: 16860