Cost-Aware Dynamic Tree Construction for Efficient Large Language Model Inference

ICLR 2026 Conference Submission18674 Authors

Keywords: Large Language Models, Speculative Decoding

TL;DR: We propose a new dynamic tree speculative decoding method that leverage the inference cost and achieves improvements against baselines.

Abstract:

Large Language Models (LLMs) face significant inference latency challenges stemming from their autoregressive design and large size. To address this, speculative decoding emerges as a solution, enabling the simultaneous generation and validation of multiple tokens. While recent approaches like EAGLE-2 and EAGLE-3 improve speculative decoding using dynamic tree structures, they often neglect the impact of crucial system variables such as GPU devices and batch sizes.

Therefore, we introduce a new dynamic tree decoding approach called CAST that takes into account inference costs, including factors such as GPU configurations and batch sizes, to dynamically refine the tree structure. Through comprehensive experimentation across six diverse tasks and utilizing six distinct LLMs, our methodology demonstrates remarkable results, achieving speeds up to 5.2 times faster than conventional decoding methods. Moreover, it generally outperforms existing state-of-the-art techniques from 5% to 20%.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 18674