Constrained Discrete Diffusion

Michael Cardei, Jacob K Christopher, Bhavya Kailkhura, Thomas Hartvigsen, Ferdinando Fioretto

Keywords: Discrete Diffusion, Controllable Generation, Differentiable Optimization

Abstract:

Discrete diffusion models are a class of generative models that construct sequences by progressively denoising samples from a categorical noise distribution. Beyond their rapidly growing ability to generate coherent natural language, these models present a new and important opportunity to enforce sequence-level constraints, a capability that current autoregressive models cannot natively provide.
This paper capitalizes on this opportunity by introducing $\textit{Constrained Discrete Diffusion}$ (CDD), a novel integration of differentiable constraint optimization within the diffusion process to ensure adherence to constraints, logic rules, or safety requirements for generated sequences. Unlike conventional text generators that often rely on post-hoc filtering or model retraining for controllable generation, CDD directly imposes constraints into the discrete diffusion sampling process, resulting in a training-free and effective approach.
Experiments in toxicity-controlled text generation, property-constrained molecule design, and instruction-constrained text completion demonstrate that CDD achieves $\textit{zero constraint violations}$ in a diverse array of tasks while preserving fluency, novelty, and coherence, and outperforming autoregressive and existing discrete diffusion approaches.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 24764