DiffuseGuide: Guiding Diffusion Models Made Easy

Nithin Gopalakrishnan Nair, Vishal M. Patel

Keywords: Diffusion models, Steering diffusion models, Automatic guidance

Abstract:

Despite advancements in conditional generation using diffusion models, conditional generation remains affected by training cost, generalizability, and speed. Training free conditional generation assists in these avenues where a model can be steered to adhere any particular condition through inference time optimization. However,
existing techniques often rely on computationally intensive backpropagation through the diffusion network to estimate the guidance direction, compounded by the need for meticulous parameter tuning tailored to individual tasks.
Although some recent works have introduced minimal-compute methods for linear inverse problems, a generic, lightweight guidance solution for both linear and non-linear guidance problems is still missing. To this end, we propose \emph{DiffuseGuide}, a method that enables inference-time guidance without compute-heavy backpropagation through the diffusion network. The key idea is to approximate the guidance direction with respect to the current sample, thereby removing the backpropagation operation. Moreover, we propose an empirical guidance scale that works for a wide variety of tasks, thus removing the need for handcrafted parameter tuning. We further introduce an effective, lightweight augmentation strategy that significantly boosts performance during inference-time guidance. We present experiments using DiffuseGuide on multiple linear and non-linear tasks across multiple datasets and models to show the effectiveness of the proposed modules.

Primary Area: generative models

Submission Number: 10124