To guide or not to guide: Improving diffusion sampling with progressive guidance

Xi WANG, Nicolas Dufour, Nefeli Andreou, Marie-Paule CANI, Victoria Fernandez Abrevaya, David Picard, Vicky Kalogeiton

Keywords: diffusion model, generative model

Abstract:

Diffusion guidance pertains to the use of conditional diffusion under specific constraints, such as class labels or textual inputs. There are primarily two methodologies: Classifier Guidance leverages a pre-trained classifier to guide generation towards the condition, while Classifier-Free Guidance achieves implicit guidance without any classifier but by entering the condition during training. Both approaches employ a weighting parameter ω that determines the trade-off between
sample fidelity (unconditional diffusion) and conditional adherence. In this paper, we posit that such conflict between image quality and condition arises, in part, due to misclassification and conflicted gradients from the explicit or implicit classifier, especially when the noise is high i.e., at the first stages of generation. To address this, we introduce a progressive weighting scheme, called Progressive-Guidance, where we make the weight of the guidance term dependent on the timestep. We propose two-time dependent weighting schemes: a simple heuristic, and a more
precise gradient-norm-based method. Progressive-Guidance can be implemented without retraining the model and with only a few additional lines of code. We report enhanced performance in benchmark metrics on three tasks: class-conditional image generation, text-to-image generation, and text-to-motion generation.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7735