Generation Properties of Stochastic Interpolation under Finite Training Set

Yunchen Li, Shaohui Lin, Zhou Yu

Keywords: Stochastic Interpolation, generative model, memory, diffusion model

TL;DR: We provide a theoretical explanation for the memorization of generative models and define the phenomena of overfitting and underfitting.

Abstract: This paper investigates the theoretical behavior of generative models under finite training populations. Within the stochastic interpolation generative framework, we derive closed-form expressions for the optimal velocity field and score function when only a finite number of training samples are available. We demonstrate that, under some regularity conditions, the deterministic generative process exactly recovers the training samples, while the stochastic generative process manifests as training samples with added Gaussian noise. Beyond the idealized setting, we consider model estimation errors and introduce formal definitions of underfitting and overfitting specific to generative models. Our theoretical analysis reveals that, in the presence of estimation errors, the stochastic generation process effectively produces convex combinations of training samples corrupted by a mixture of uniform and Gaussian noise. Experiments on generation tasks and downstream tasks such as classification support our theory.

Primary Area: generative models

Submission Number: 2746