SeeD: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding
Abstract: Large Language Models (LLMs) demonstrate remarkable emergent abilities across various tasks, yet fall short of complex reasoning and planning tasks.
The tree-search-based reasoning methods address this by surpassing the capabilities of chain-of-thought prompting, encouraging exploration of intermediate steps.
However, such methods introduce significant inference latency due to the systematic exploration and evaluation of multiple thought paths.
This paper introduces SeeD, a novel and efficient inference framework to optimize runtime speed and GPU memory management concurrently.
By employing a scheduled speculative execution, SeeD efficiently handles multiple iterations for the thought generation and the state evaluation, leveraging a rounds-scheduled strategy to manage draft model dispatching.
Extensive experimental evaluations on three reasoning datasets demonstrate superior speedup performance of SeeD, providing a viable path for batched inference in training-free speculative decoding.
Paper Type: Long
Research Area: Special Theme (conference specific)
Research Area Keywords: efficiency in inference
Contribution Types: Approaches low compute settings-efficiency
Languages Studied: English
Submission Number: 2366
Loading