Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models

Benjamin Dupuis, Dario Shariatian, Maxime Haddouche, Alain Oliviero Durmus, Umut Simsekli

Keywords: Diffusion models, Generalization bounds, Score-based generative models

TL;DR: We prove algorithm- and data-dependent upper bounds on the generalization error of diffusion models by using tools from statistical learning theory

Abstract: Score-based generative models (SGMs) have emerged as one of the most popular classes of generative models. A substantial body of work now exists on the analysis of SGMs, focusing either on discretization aspects or on their statistical performance. In the latter case, bounds have been derived, under various metrics, between the true data distribution and the distribution induced by the SGM, often demonstrating polynomial convergence rates with respect to the number of training samples. However, these approaches adopt a largely approximation theory viewpoint, which tends to be overly pessimistic and relatively coarse. In particular, they fail to fully explain the empirical success of SGMs or capture the role of the optimization algorithm used in practice to train the score network. To support this observation, we first present simple experiments illustrating the concrete impact of optimization hyperparameters on the generalization ability of the generated distribution. Then, this paper aims to bridge this theoretical gap by providing the first algorithmic- and data-dependent generalization analysis for SGMs. In particular, we establish bounds that explicitly account for the optimization dynamics of the learning algorithm, offering new insights into the generalization behavior of SGMs. Our theoretical findings are supported by empirical results on several datasets.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 12063