Data Prediction Denoising Models: The Pupil Outdoes the Master
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Diffusion Model, Generative Models
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Due to their flexibility, scalability, and high quality,
diffusion models (DMs) have become a fundamental stream of modern AIGC. However, a substantial performance deficit of DMs emerges when confronted with a scarcity of sampling steps. This limitation stems from the DM's acquisition of a series of weak denoisers obtained by minimizing a denoising auto-encoder objective. The weak denoisers lead to a decline in the quality of generated data samples in scenarios with few sampling steps. To address this, in this work, we introduce the Data-Prediction Denoising Model (DPDM), a constructor that embodies a sequence of stronger denoisers compared to conventional diffusion models. The DPDM is trained by initializing from a teacher DM. The core idea of training DPDM lies in improving the denoisers' data recovery ability with noisy data as inputs. We formulate such an idea through the minimization of suitable probability divergences between denoiser-recovered data distributions and the ground truth data distribution. The sampling algorithm of the DPDM is executed through an iterative process that interleaves data prediction and the sequential introduction of noise. We conduct a comprehensive evaluation of the DPDM on two tasks: data distribution recovery and the few-step image data generation. For the data distribution recovery, the DPDM shows significantly stronger ability to recover data distributions from noisy distribution. For the data generation task, we train DPDMs on two benchmark datasets: the CIFAR10, and the ImageNet$64\times 64$. We compare the DPDM with baseline diffusion models together with other diffusion-based multi-step generative models under the few-step generation setting. We observe the superior performance advantage of DPDMs over competitor methods. In addition to the strong empirical performance, we also elucidate the interconnections and comparisons between the DPDM and existing methodologies, which shows that DPDM is a stand-alone generative model that is essentially different from existing models.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5134
Loading