Momentum-driven Noise-free Guided Conditional Sampling for Denoising Diffusion Probabilistic Models
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: diffusion models, conditional image generation, arbitrary style-transfer
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: We present a novel approach for conditional sampling of denoising diffusion probabilistic models (DDPM) using noise-free guidance, which is applicable to any guidance function operating on clean data. We observe that the performance gap between previous clean estimation $(\widehat{x_0})$-based methods and noised sample $(x_t)$-based methods stems from the incorporation of estimation deviation in the clean-estimation guidance process. The former contrasts with noise-guided techniques where noise contamination is addressed by a noise-finetuned classifier, leading to inconsistent and unreliable guidance gradients from the inaccurate clean estimation. To tackle this issue, we propose a two-fold solution: (1) implementing momentum-driven gradient filtering to stabilize the gradient transmitted from the guidance function, ensuring coherence throughout the denoising process, and adaptively adjusting the update stepsize of pivot pixels to increase their resilience against detrimental gradients; and (2) introducing a guidance suppression scheme to alleviate the impact of unreasonably large weights assigned considering the significantly larger estimation deviation in the early stage. Our momentum-driven noise-free conditional sampling method demonstrates superior performance in guided image generation tasks at less than 2\% of the training cost for classifiers, eliminating the need for noise-finetuning. Moreover, it offers the potential for reusing guidance on DDPM with other noise schedules. We further showcase the versatility of our approach by applying it to the arbitrary style transfer task, achieving state-of-the-art performance without being limited to labeled datasets.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4280
Loading