DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure
Keywords: inference methods, efficient inference, speculative decoding
Abstract: While speculative decoding has recently appeared as a promising direction for accelerating the inference of large language models (LLMs), the speedup and scalability are strongly bounded by the token acceptance rate.
Prevalent methods usually organize predicted tokens as independent chains or fixed token trees, which fails to generalize to diverse query distributions.
In this paper, we propose \textsc{DySpec}, a faster speculative decoding algorithm with a novel dynamic token tree structure.
We begin by bridging the draft distribution and acceptance rate from
intuitive and empirical clues, and successfully show that the two variables are strongly correlated. Based on this, we employ a greedy strategy to dynamically expand the token tree at run time. Theoretically, we show that our method can achieve optimal results under mild assumptions. Empirically, \textsc{DySpec} yields a higher acceptance rate and speedup than fixed trees. \textsc{DySpec} can drastically improve the throughput and reduce the latency of token generation across various data distribution and model sizes, which significantly outperforms strong competitors, including Specinfer and Sequoia. Under low temperature setting, \textsc{DySpec} can improve the throughput up to 9.10x and reduce the latency up to 9.4x on Llama2-70B. Under high temperature setting, \textsc{DySpec} can also improve the throughput up to 6.21x, despite the increasing difficulty of speculating more than one token per step for draft model.
Primary Area: infrastructure, software libraries, hardware, systems, etc.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 14236
Loading