Democratized Diffusion Language Model

Sofia Maria Lo Cicero Vaina, Nikita Balagansky, Daniil Gavrilov

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Diffusion LMs, Language Modelling, Early Exiting, Diffusion Early Exiting

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We observed that Diffusion LMs can halter the generation process and facilitate an adaptive early exit.

Abstract: Diffusion Models are a promising avenue for text generation, offering a multitude of frameworks for researchers and practitioners alike. These frameworks differ based on how the Diffusion Model is utilized for categorical data generation. This paper aims to look into these differences by examining the SSD and Plaid models, as well as our attentive replication of the CDCD models. Our study focuses mainly on the process of text generation performed at runtime by various frameworks. One of our research's notable findings is that, according to our observations, most models are capable of halting the generation process and facilitating an adaptive early exit. This feature proves instrumental in accelerating the speed of text generation by Diffusion Language Models without compromising the quality of the generated text.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2395