RepSpec: Structural Re-parameterized Draft Model Training for Speculative Decoding

ICLR 2026 Conference Submission16547 Authors

Keywords: Reparameterization, Speculative Decoding

TL;DR: We introduce RepSpec, a training method for speculative decoding that uses structural re-parameterization to temporarily expand the draft model’s capacity during training—without adding inference cost.

Abstract: As the parameter size of large language models (LLMs) continues to grow, the latency of autoregressive inference increases due to memory-bound computational inefficiency. To address this, speculative decoding has been proposed, where a large target model verifies multiple tokens generated in parallel by a smaller draft model. However, the performance of speculative decoding is fundamentally limited by the draft model’s capacity, which stems from the parameter gap between the two models. To overcome this limitation, we propose RepSpec, which combines structural re-parameterization with draft model training. During training, redundant linear structures are introduced and later merged into the backbone network during inference, thus enhancing the draft model’s training effectiveness without increasing inference cost. By applying our method to improve the current state-of-the-art approach, EAGLE, we achieve a significant improvement in accepted sequence length. Furthermore, considering the specific characteristics of the speculative decoding scenario, we explore a hybrid training strategy that combines linear and nonlinear structures, which yields a further improvement in acceptance length.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 16547