Improved Distribution Matching Distillation for Fast Image Synthesis
Keywords: Image Generation, diffusion based models, model distillation
TL;DR: We distill diffusion models into few-step generators that produce images with superior quality.
Abstract: Recent approaches have shown promises distilling expensive diffusion models into efficient one-step generators.
Amongst them, Distribution Matching Distillation (DMD) produces one-step generators that match their teacher in distribution, i.e., the distillation process does not enforce a one-to-one correspondence with the sampling trajectories of their teachers.
However, to ensure stable training in practice, DMD requires an additional regression loss computed using a large set of noise--image pairs, generated by the teacher with many steps of a deterministic sampler.
This is not only computationally expensive for large-scale text-to-image synthesis, but it also limits the student's quality, tying it too closely to the teacher's original sampling paths.
We introduce DMD2, a set of techniques that lift this limitation and improve DMD training.
First, we eliminate the regression loss and the need for expensive dataset construction.
We show that the resulting instability is due to the "fake" critic not estimating the distribution
of generated samples with sufficient accuracy and propose a two time-scale update rule as a remedy.
Second, we integrate a GAN loss into the distillation procedure, discriminating between generated samples and real images.
This lets us train the student model on real data, thus mitigating the imperfect "real" score estimation from the teacher model, and thereby enhancing quality.
Third, we introduce a new training procedure that enables multi-step sampling in the student, and
addresses the training--inference input mismatch of previous work, by simulating inference-time generator samples during training.
Taken together, our improvements set new benchmarks in one-step image generation, with FID scores of 1.28 on ImageNet-64×64 and 8.35 on zero-shot COCO 2014, surpassing the original teacher despite a 500X reduction in inference cost.
Further, we show our approach can generate megapixel images by distilling SDXL, demonstrating exceptional visual quality among few-step methods, and surpassing the teacher.
We release our code and pretrained models.
Primary Area: Diffusion based models
Submission Number: 1257
Loading