Diffusion models for Gaussian distributions: Exact solutions and Wasserstein errors

Emile Pierret, Bruno Galerne

TL;DR: A theoretical study of the convergence of diffusion models for Gaussian distributions.

Abstract:

Diffusion or score-based models recently showed high performance in image generation.
They rely on a forward and a backward stochastic differential equations (SDE). The sampling of a data distribution is achieved by numerically solving the backward SDE or its associated flow ODE.
Studying the convergence of these models necessitates to control four different types of error: the initialization error, the truncation error, the discretization error and the score approximation.
In this paper, we theoretically study the behavior of diffusion models and their numerical implementation when the data distribution is Gaussian.
Our first contribution is to derive the analytical solutions of the backward SDE and the probability flow ODE and to prove that these solutions and their discretizations are all Gaussian processes.
Our second contribution is to compute the exact Wasserstein errors between the target and the numerically sampled distributions for any numerical scheme.
This allows us to monitor convergence directly in the data space, while experimental works limit their empirical analysis to Inception features.
An implementation of our code is available online.

Lay Summary:

Diffusion models, a type of machine learning model, have recently become very good at generating images. These models work by running a process forward to add noise to data, and then reversing it to recover or generate new data. This forward and backward process is described using mathematical tools called stochastic differential equations (SDEs).
To understand how well these models work, we need to look at four types of errors that can happen during this process: errors at the start (initialization), from stopping too early (truncation), from using step-by-step calculations (discretization), and from estimating certain model parts (score approximation).
In this paper, we focus on a simpler case where the data follows a Gaussian distribution, which allows us to study the math more precisely. Our first main result is that we find exact mathematical solutions for the reverse process used in diffusion models and show that both the original and approximated versions behave like Gaussian processes.
Our second main result is that we calculate the exact differences (using a metric called Wasserstein distance) between the ideal results and the ones produced by the numerical simulations. This helps us measure how well the model is working.

Link To Code: https://github.com/emilePi/Diffusion-models-for-Gaussian-distributions-Exact-solutions-and-Wasserstein-errors

Primary Area: Theory->Probabilistic Methods

Keywords: Convergence of diffusion models, Gaussian processes, Wasserstein distance, stochastic differential equations, discretization schemes

Submission Number: 10442