Stochastic Adaptive Sequential Black-box Optimization for Diffusion Targeted Generation

Yueming Lyu, Kim Yong Tan, Yew-Soon Ong, Ivor Tsang

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Black-box Optimization, Diffusion Model, Targeted Generation

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Diffusion models have demonstrated great potential in generating high-quality content for images, natural language, protein domains, etc. However, how to perform user-preferred targeted generation via diffusion models with only black-box target scores of users remains challenging. To address this issue, we first formulate the fine-tuning of the inference phase of a pre-trained diffusion model as a sequential black-box optimization problem. Furthermore, we propose a novel stochastic adaptive sequential optimization algorithm to optimize cumulative black-box scores under unknown transition dynamics. Theoretically, we prove a $O(\frac{d^2}{\sqrt{T}})$ convergence rate for cumulative convex functions without smooth and strongly convex assumptions. Empirically, we can naturally apply our algorithm for diffusion black-box targeted generation. Experimental results demonstrate the ability of our method to generate target-guided images with high target scores.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4372