Analyzing Neural Network Based Generative Diffusion Models via Convexification

Fangzhao Zhang, Mert Pilanci

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: diffusion; score matching; convex optimization;

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We derive a convex program for score matching objective and study some of its properties.

Abstract:

Diffusion models are becoming widely used in state-of-the-art image, video and audio generation. Score-based diffusion models stand out among these methods, necessitating the estimation of the score function of the input data distribution. In this study, we present a theoretical framework to analyze two-layer neural network-based diffusion models by reframing score matching and denoising score matching as convex optimization. We show that the global optimum of the score matching objective can be attained by solving a simple convex program. Specifically, for univariate training data, we establish that the Langevin diffusion process through the learned neural network model converges in the Kullback-Leibler (KL) divergence to either a Gaussian or a Gaussian-Laplace distribution when the
weight decay parameter is set appropriately. Our convex programs alleviate issues in computing the Jacobian and also extends to multidimensional score matching.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8307