Dynamic Prompt Evolution via Multi-Attribute Feedback for Text-to-Image Generation

Shuo Liu, Leizhang, Liyan Wang, Limali, cheng yang, Zhixun Su, Wenhao Jiang

Keywords: Multi-Attribute Feedback, Zero-Order Optimization, Prompt Evolution

TL;DR: An Optimization Method for Input Prompts of Diffusion Models

Abstract: Most existing text-to-image methods primarily focus on enhancing model comprehension or tuning prompt strength, while overlooking the intrinsic expressive limitations of the prompts themselves—particularly in aligning the generated content with user-specified attributes. In this paper, we propose \textit{Dynamic Prompt Evolution via Multi-Attribute Feedback} (DPE-MAF), a framework that integrates heuristic algorithms with the prior knowledge of large language models (LLMs) to dynamically generate optimal prompts tailored to user-intended attributes for text-to-image generation.Specifically, we formulate the prompt optimization task as a zero-order optimization problem within the natural language space. To address this, we introduce a \textit{Diffusion-Heuristic Optimization} (DHO) module, which employs LLMs to expand an initial prompt into a candidate population and performs heuristic iterative search guided by multiple attributes, thereby dynamically steering the natural language optimization toward user-desired outputs.To further guide the iterative process, we propose a \textit{Dynamic Contrastive Guidance Update} (DCGU) mechanism, which refines the search by contrasting semantic features between high-quality and low-quality prompts. This contrastive feedback facilitates convergence toward the global optimum. Our approach significantly enhances the ability of diffusion-based generative models to produce semantically consistent and high-fidelity images in text-to-image tasks. Extensive experiments demonstrate that DPE-MAF effectively evolves prompts to improve the performance of various diffusion models, surpassing state-of-the-art prompt-based approaches in text-to-image generation.

Primary Area: generative models

Submission Number: 10384