Revision History for Towards Training One-Step Diffusion...

Camera Ready Edit by Authors

• 28 Mar 2025, 17:45 Coordinated Universal Time

• Title: Towards Training One-Step Diffusion Models Without Distillation
• Authors: Mingtian Zhang, Jiajun He, Wenlin Chen, Zijing Ou, José Miguel Hernández-Lobato, Bernhard Schölkopf, David Barber
• Track: tiny / short paper (up to 4 pages)
• Authorids: Mingtian Zhang, Jiajun He, Wenlin Chen, Zijing Ou, José Miguel Hernández-Lobato, Bernhard Schölkopf, David Barber
• Keywords: Diffusion model, one-step generative model, distillation, GAN
• TLDR: We investigate training one-step generative models without traditional distillation, showing that score estimation is unnecessary while teacher weight initialization is crucial for feature transfer, shaping distillation quality.
• Abstract:

  Recent advances in one-step generative models typically follow a two-stage process: first training a teacher diffusion model and then distilling it into a one-step student model. This distillation process traditionally relies on both the teacher model's score function to compute the distillation loss and its weights for student initialization. In this paper, we explore whether one-step generative models can be trained directly without this distillation process. First, we show that the teacher's score function is not essential and propose a family of distillation methods that achieve competitive results without relying on score estimation. Next, we demonstrate that initialization from teacher weights is indispensable in successful training. Surprisingly, we find that this benefit is not due to improved ``input-output" mapping but rather the learned feature representations, which dominate distillation quality. Our findings provide a better understanding of the role of initialization in one-step model training and its impact on distillation quality.

• PDF: pdf