AccCtr: Accelerating Training-Free Control For Text-to-Image Diffusion Models
Keywords: accelerating, Training-Free, diffusion model
Abstract: In training-free Conditional Diffusion Models (CDMs), the sampling process is steered by the gradient of the loss $\mathcal{E}(\dmrv{y}, \dmrv{z}, \dmfv{C}_{\dmv{psi}} )$, which assesses the gap between the guidance $\dmrv{y}$ and the condition extracted from the intermediate outputs. Here the condition extraction network $\dmfv{C}_{\dmv{psi}}(\cdot)$, which could be a segmentation or depth estimation network, is pre-trained for training-free purpose. However, existing methods often require small guidance steps, leading to longer sampling times. We introduce an alternative maximization framework to scrutinize training-free CDMs that tackles slow sampling. Our framework pinpoints manifold deviation as the key factor behind the sluggish sampling. More iterations are needed for the sampling process to closely follow the image manifold and reach the target conditions, as the loss gradient doesn't provide sufficient guidance for larger steps. To improve this, we suggest retraining the condition extraction network $\dmfv{C}_{\dmv{psi}}(\cdot)$ to refine the loss's guidance, thereby introducing our AccCtr. This retraining process is simple, and integrating AccCtr into current CDMs is a seamless task that does not impose a significant computational burden. Extensive testing has demonstrated that AccCtr significantly boosts performance, offering superior sample quality and faster generation times across a variety of conditional generation tasks.
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 13041
Loading