DSSD: Efficient Edge-Device Deployment and Collaborative Inference via Distributed Split Speculative Decoding

JIAHONG NING, Ce ZHENG, Tingting Yang

Keywords: speculative decoding, edge-device deployment, collaborative inference

Abstract: Large language models (LLMs) have transformed natural language processing but face critical deployment challenges in device-edge-cloud ecosystems due to resource limitations and communication overhead. To address these issues, collaborative frameworks have emerged that combine small language models (SLMs) on devices with LLMs at the edge, using speculative decoding (SD) to improve efficiency. However, existing solutions often trade inference accuracy for latency or suffer from high uplink transmission costs when verifying candidate tokens. In this paper, we propose Distributed Split Speculative Decoding (DSSD), a novel architecture.

Submission Number: 34