Understanding Generalization in Diffusion Distillation via Probability Flow Distance

Huijie Zhang, Zijian Huang, Siyi Chen, Jinfan Zhou, Zekai Zhang, Peng Wang, Qing Qu

Keywords: generalization, diffusion model, distillation, metric, probability flow distance

Abstract:

Diffusion distillation provides an effective approach for learning lightweight and few-steps diffusion models with efficient generation.
However, evaluating their generalization remains challenging: theoretical metrics are often impractical for high-dimensional data, while no practical metrics rigorously measure generalization.
In this work, we bridge this gap by introducing probability flow distance ($\mathtt{PFD}$), a theoretically grounded and computationally efficient metric to measure generalization.
Specifically, $\mathtt{PFD}$ quantifies the distance between distributions by comparing their noise-to-data mappings induced by the probability flow ODE.
Using $\mathtt{PFD}$ under the diffusion distillation setting, we empirically uncover several key generalization behaviors, including:
(1) quantitative scaling behavior from memorization to generalization,
(2) epoch-wise double descent training dynamics, and
(3) bias-variance decomposition. Beyond these insights, our work lays a foundation for generalization studies in diffusion distillation and bridges them with diffusion training.

Submission Number: 50