S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUs
Abstract: Speculative decoding (SD) has attracted a significant amount of research attention due to the substantial speedup it can achieve for LLM inference.
However, speculative decoding methods often achieve optimal performance on high-end devices or with a substantial GPU memory overhead.
Given limited memory and the necessity of quantization, a high-performing SD model on a high-end GPU can slow down by up to 7 times.
To this end, we propose Skippy Simultaneous Speculative Decoding (or S3D), a cost-effective self-speculative SD method based on simultaneous multi-token decoding and mid-layer skipping.
When compared against recent effective open-source SD systems, our method has achieved one of the top performance-memory ratios while requiring minimal architecture changes and training data.
Leveraging our memory efficiency, we created a smaller yet more effective SD model based on Phi-3. It is 1.4 to 2 times faster than the quantized EAGLE model and operates in half-precision while using less VRAM.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: Speculative decoding, memory efficiency, layer skipping, multi-token predictions.
Contribution Types: Approaches to low-resource settings, Approaches low compute settings-efficiency
Languages Studied: English
Submission Number: 231
Loading