High variance score function estimates help diffusion models generalize

John Vastola

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: generative modeling, score-based modeling, score matching, generalization, diffusion, theory

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: The denoising score matching objective used to train diffusion models produces extremely noisy small-time score function estimates in a way that helps diffusion models generalize.

Abstract: How do diffusion-based generative models generalize beyond their training set? In particular, do they perform something similar to kernel density estimation? If so, what is the kernel, and which aspects of training and sampling determine its form? We argue that a key contributor to generalization is the fact that the denoising score matching objective usually used to train diffusion models tends to obtain high variance score function estimates at early times. We investigate this claim by mathematically studying (unconditional) diffusion models in a variety of analytically tractable settings (e.g., when the training distribution is a Gaussian mixture), and are able to compute various exact and asymptotic expressions for quantities like the variance of score function parameter estimates. We show that the effect of this high variance is mathematically equivalent to running reverse diffusion using the "optimal" score, and then convolving the result with a data-dependent kernel function.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8893