Speculative Speculative Decoding

Tanishq Kumar, Tri Dao, Avner May

Keywords: inference, large language models, speculative decoding

TL;DR: We introduce an asynchronous speculative decoding algorithm wherein the draft model continuously speculates on top of anticipated verification outcomes, thus hiding drafting latency entirely.

Abstract: Autoregressive decoding is bottlenecked by its *sequential* nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them *in parallel* with a single target model forward pass. However, speculative decoding itself relies on a *sequential* dependence between speculation and verification. We introduce *speculative speculative decoding* (SSD) to parallelize these operations. While a verification is ongoing, the draft model *predicts* likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is up to 2x faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 9487