Consistent Diffusion Language Models

ICLR 2026 Conference Submission20760 Authors

Keywords: Diffusion Language Models, Discrete Diffusion Models, Consistency Models, Generative Modeling

Abstract: Diffusion-based language models (DLMs) have emerged as compelling alternatives to sequential autoregressive generation, offering the promise of parallel decoding. Yet existing discrete diffusion models require hundreds of refinement steps for high-quality text, undermining the efficiency gains of parallelism. We introduce the Consistent Diffusion Language Model (CDLM), a new family of generative models that brings the benefits of consistency training---enforcing agreement across noise levels to enable one- or few-step generation---to the discrete domain. Our approach leverages an exact closed-form formulation of discrete posteriors, providing a rigorous analogue to the missing probability-flow ODE in discrete space. This yields a multi-path consistency objective that, as we show, unifies and generalizes popular diffusion, consistency, and distillation methods in a single view. To ensure stability at scale, we introduce a set of principled design choices that prevent training pathologies like mode collapse. On conditional and unconditional text-generation benchmarks, CDLM establishes new state of the art as a single-stage model, consistently outperforming both base and distilled DLMs across sampling budgets. These results position CDLM as a new paradigm for efficient, scalable, and high-fidelity discrete generative modeling. We will be updating the code base under https://anonymous.4open.science/r/dlm-135B

Primary Area: generative models

Submission Number: 20760