Towards Latent Diffusion Suitable For Text

Nesta Midavaine, Christian A. Naesseth, Grigory Bartosh

Keywords: diffusion LM, generative models

TL;DR: We extend Neural Flow Diffusion Models to language, enabling efficient continuous diffusion in discrete text spaces through a learned, data-driven forward process well-suited to language generation.

Abstract: Language diffusion models aim to improve sampling speed and coherence over autoregressive LLMs. We introduce Neural Flow Diffusion Models for language generation, an extension of NFDM that enables the straightforward application of continuous diffusion models to discrete state spaces. NFDM learns a multivariate forward process from the data, ensuring that the forward process and generative trajectory are a good fit for language modeling. Our model substantially reduces the likelihood gap with autoregressive models of the same size, while achieving sample quality comparable to that of previous latent diffusion models. The code is available at https://github.com/Nesta-gitU/discrete_diffusion.

Submission Number: 21