SpecCoT: Accelerating Chain-of-Thought Reasoning through Speculative Exploration

Junhan Shi, Yijia Zhu, Zhenning Shi, Dan Zhao, Qing Li, Yong Jiang

Keywords: efficient reasoning, chain-of-thought, speculative decoding

Abstract: Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks through chain-of-thought (CoT) reasoning. However, they suffer from high inference latency due to lengthy reasoning chains and the substantial computational requirements of large models. In this paper, we propose SpecCoT, a collaborative framework that combines large and small models for effective yet efficient reasoning. Unlike traditional speculative decoding, which operates at the token level, SpecCoT adopts a step-level verification strategy: the large model first establishes the reasoning direction, and for each intermediate step, the small model generates multiple candidate drafts in parallel. The large model then verifies these drafts, either selecting the most suitable one or rejecting them all and generating its own response. This SpecCoT approach balances reasoning quality with inference efficiency through fine-grained model cooperation. Experiments across diverse tasks show that SpecCoT reduces inference latency by 1.7-4.1$\times$ while maintaining comparable accuracy to standard large model inference.

Submission Number: 92