SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decoding

Yijun Lin, Jinhao Sheng, Qingyue Cai, Feng Zhou

Keywords: Speculative Decoding, Multi-Draft, Block Verification, Inference Efficiency

TL;DR: This paper introduces SpecTr-GBV, a speculative decoding framework that unifies multi-draft and block verification to achieve optimal efficiency and faster inference in autoregressive language models.

Abstract: Autoregressive language models achieve state-of-the-art performance across a wide range of natural language processing tasks, but suffer from high inference latency due to their sequential decoding nature. Speculative decoding (SD) mitigates this by employing a lightweight draft model to propose candidate tokens, which are selectively verified by a larger target model. While existing methods either adopt multi-draft strategies to increase acceptance rates or block verification techniques to jointly verify multiple tokens, they remain limited by treating these improvements in isolation. In this work, we propose SpecTr-GBV, a novel SD method that unifies multi-draft and greedy block verification (GBV) into a single framework. By formulating the verification step as an optimal transport problem over draft and target token blocks, SpecTr-GBV improves both theoretical efficiency and empirical performance. We theoretically prove that SpecTr-GBV achieves the optimal expected number of accepted tokens for any fixed number of draft sequences, and this bound improves as the number of drafts increases. Empirically, we evaluate SpecTr-GBV across five datasets and four baselines. Our method achieves superior speedup and significantly higher block efficiency while preserving output quality. In addition, we perform comprehensive ablation studies to evaluate the impact of various components in the model.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 10164