Information-Theoretic DiffusionDownload PDF

Published: 01 Feb 2023, 19:23, Last Modified: 01 Feb 2023, 19:23ICLR 2023 posterReaders: Everyone
Keywords: diffusion, density models, information theory
TL;DR: A new information-theoretic foundation for diffusion models leads to simpler and more computationally efficient density modeling.
Abstract: Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation. Whether interpreted through the lens of variational models or differential equations, diffusion models require many steps of expensive computation to give accurate density estimates. We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connects Information with Minimum Mean Square Error estimators, the so-called I-MMSE relations. We generalize the I-MMSE relations to \emph{exactly} relate the data distribution and optimal denoising, leading to an elegant refinement of existing diffusion bounds. This new insight improves density estimation for diffusion models and enables simultaneous modeling of both continuous and discrete probabilities with no additional cost.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Supplementary Material: zip
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Submission Guidelines: Yes
Please Choose The Closest Area That Your Submission Falls Into: Unsupervised and Self-supervised learning
13 Replies