PLM-Based Discrete Diffusion Language Models with Entropy-Adaptive Gibbs Sampling

ACL ARR 2024 December Submission2058 Authors

Abstract:

Recently, discrete diffusion language models have demonstrated promising results in NLP.
However, there has been limited research on integrating Pretrained Language Models (PLMs) into discrete diffusion models, resulting in underwhelming performance in downstream NLP generation tasks.
This integration is particularly challenging because of the discrepancy between step-wise denoising strategy of diffusion models and single-step mask prediction approach of MLM-based PLMs.
In this paper, we introduce Diffusion-EAGS, a novel approach that effectively integrates PLMs with the diffusion models.
Furthermore, as it is challenging for PLMs to determine where to apply denoising during the diffusion process, we integrate an entropy tracking module to assist them. Finally, we propose entropy-based noise scheduling in the forward process to improve the effectiveness of entropy-adaptive sampling throughout the generation phase.
Experimental results show that Diffusion-EAGS outperforms existing diffusion baselines in downstream generation tasks, achieving high text quality and diversity with precise token-level control. We also show that our model is capable of adapting to bilingual and low-resource settings, which are common in real-world applications.

Paper Type: Long

Research Area: Generation

Research Area Keywords: Model architectures, Language Modeling

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models, Theory

Languages Studied: english, german

Submission Number: 2058