Not All Noises Are Created Equally: Diffusion Noise Selection and Optimization
Keywords: Diffusion Models, Optimization, Generative AI, Text-to-Image Generation
TL;DR: We report that noise selection and optimization via inversion stability can significantly improve generated results of diffusion models over random Gaussian noises in a a plug-and-play manner.
Abstract: Diffusion models that can generate high-quality data from randomly sampled Gaussian noises have become the mainstream generative method in academia and industry. Are randomly sampled Gaussian noises equally effective for diffusion models? Some methods explore the impact of noise variations on the results, but they either do not operate in the pure noise space, requiring additional evaluation models, or cannot be adapted to general text-to-image tasks. In this paper, we mainly made three contributions. First, we are the first to hypothesize and empirically observe that the generation quality of diffusion models significantly depends on the noise inversion stability. This naturally provides a noise quality metric for noise selection, grounded in a mathematical property. Second, we further propose a novel noise optimization method that actively enhances the inversion stability of arbitrary given noises. Our method is the first one that works purely on noise space for general text-to-image without fine-tuning model parameters or relying on additional result quality evaluators. Third, our extensive experiments demonstrate that the proposed noise selection and noise optimization methods both significantly improve representative diffusion models, such as SDXL and SDXL-turbo, in terms of human preference and other objective evaluation metrics. For example, the human preference winning rates of noise selection and noise optimization over the baselines can be up to 57% and 72.5%, respectively, on DrawBench.
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5555
Loading