FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion

ICLR 2026 Conference Submission18874 Authors

Keywords: Diffusion Model, Diffusion Language Model, Inference Speed, Language Models, Efficient Inference

Abstract:

Diffusion language models offer parallel token generation and inherent bidirectionality, promising more efficient and powerful sequence modeling compared to autoregressive approaches. However, state-of-the-art diffusion models~(e.g., Dream 7B, LLaDA 8B) suffer from slow inference. While they match the quality of similarly sized Autoregressive (AR) Models (e.g., Qwen2.5 7B, Llama3 8B), their iterative denoising requires multiple full-sequence forward passes, resulting in high computational costs and latency, particularly for long input prompts and long-context scenarios. Furthermore, parallel token generation introduces token incoherence problems, and current sampling heuristics suffer from significant quality drops with decreasing denoising steps. We address these limitations with two training-free techniques. First, we propose \texttt{FreeCache}, a Key-Value (KV) approximation caching technique that reuses stable KV projections across denoising steps, effectively reducing the computational cost of DLM inference.
Second, we introduce \texttt{Guided Diffusion}, a training-free method that uses a lightweight pretrained autoregressive model to supervise token unmasking, dramatically reducing the total number of denoising iterations without sacrificing quality.
We conduct extensive evaluations on open-source reasoning benchmarks, and our combined methods deliver an average of \textbf{12.14}$\times$ end-to-end speedup across various tasks with negligible accuracy degradation. For the first time, diffusion language models achieve a comparable and even faster latency as the widely adopted autoregressive models. Our work successfully paved the way for scaling up the diffusion language model to a broader scope of applications across different domains. Our code and implementation are anonymously available at https://anonymous.4open.science/r/anon-flash-dlm-A42B/.

Primary Area: generative models

Submission Number: 18874