Highlight Diffusion: Training-Free Attention Guided Acceleration for Text-to-Image Models
Keywords: Diffusion model, Cross attention, Acceleration, Text to Image
TL;DR: Accelerating diffusion models in text-to-image by partially computing highlighted regions identified by the cross attention map
Abstract: Diffusion models have achieved exceptional results in image synthesis, yet their sequential processing nature imposes significant computational demands and latency, posing challenges for practical deployment. In this paper, we present Highlight Diffusion: a training-free novel acceleration approach that achieves significant speedup while retaining generation quality through an attention-guided generation process. By utilizing cross-attention maps to identify crucial segments within the image, we selectively compute these highlighted regions during the denoising process, bypassing the need for full-resolution computation at every step. This strategy maintains high-quality outputs while enabling faster, more resource-efficient diffusion model inference. With minimal loss in generated image quality—evidenced by only a 0.65 increase in FID score and a 0.02 decrease in CLIP score, Highlight Diffusion achieved a 1.52 $\times$ speedup using an NVIDIA RTX 3090 GPU.
Supplementary Material: zip
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 13764
Loading