The Convergence of Variance Exploding Diffusion Models under the Manifold Hypothesis

Ruofeng Yang, Zhijie Wang, Bo Jiang, Shuai Li

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Convergence guarantee, Variance Exploding Diffusion Models, Manifold Hypothesis

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Variance Exploding (VE) based diffusion models, an important class of diffusion models, have empirically shown state-of-the-art performance in many tasks. However, there are only a few theoretical works on the VE-based models, and those works suffer from a worse convergence rate $1/\text{poly}(T)$ than the $\exp{(-T)}$ results of Variance Preserving (VP) based models. The slow convergence rate is due to the Brownian Motion without the drift term and introduces hardness in balancing the different error sources. In this work, we design a new forward VESDE process with a small drift term, which converts data into pure Gaussian noise while the variance explodes. Furthermore, unlike the previous theoretical works, we allow the diffusion coefficient to be unbounded instead of a constant, which is closer to the SOTA VE-based models. With an aggressive diffusion coefficient, the new forward process allows a faster $\exp{(-T)}$ rate. By exploiting this new forward process, we prove the first polynomial sample complexity for VE-based models with reverse SDE under the realistic manifold hypothesis. Then, we focus on a more general setting considering reverse SDE and probability flow ODE simultaneously and propose the unified tangent-based analysis framework for VE-based models. In this framework, we prove the first quantitative convergence guarantee for SOTA VE-based models with probability flow ODE.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5363