Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Jungwoo Kim, Wooseok Jang, SeonHwa Kim, Hyunhee Park, Kyong Hwan Jin, Seungryong Kim

Abstract: Diffusion models can generate high-quality samples, but their quality is highly reliant on guidance techniques such as classifier guidance (CG) and classifier-free guidance (CFG), which are inapplicable in unconditional generation. Inspired by the semantic awareness capabilities of self-attention mechanisms, we present Perturbed-Attention Guidance (PAG), a method that enhances the structure of generated samples. This is done by creating degraded output through substituting the self-attention map with an identity matrix so that sampling process can be guided with those samples. As a result, in both ADM and Stable Diffusion, PAG surprisingly improves sample quality in conditional and even unconditional scenarios without additional training. Moreover, PAG significantly improves the performance in downstream tasks where existing guidance cannot be fully utilized, such as inverse problems (super-resolution, deblurring, etc.) and ControlNet with empty prompts.