Conditional [MASK] Discrete Diffusion Language Model

ACL ARR 2025 February Submission3324 Authors

Abstract: Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability. Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation. To address these challenges, we propose ***Diffusion-EAGS***, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a ***conditional Markov Random Field***. In doing so, we propose ***entropy-adaptive Gibbs sampling*** and ***entropy-based noise scheduling*** to counterbalance each model’s shortcomings. Experimental results show that ***Diffusion-EAGS*** outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation.

Paper Type: Long

Research Area: Generation

Research Area Keywords: model architectures

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study, Theory

Languages Studied: english

Submission Number: 3324