Plug-and-Play Prompt Refinement via Latent Feedback for Diffusion Model Alignment

Suhyeon Lee, Jong Chul Ye

Keywords: Diffusion Models, Reinforcement Learning, Reward Alignment, Prompt Refinement, Latent Feedback

TL;DR: PromptLoop is a PnP framework that achieves structural equivalence to parameter-level fine-tuning by aligning diffusion models through RL-tuned prompt refinement with step-wise latent feedback, while preserving generalization and composability.

Abstract: Despite the recent progress, reinforcement learning (RL)-based fine-tuning of diffusion models often struggles with generalization, composability, and robustness against reward hacking. Recent studies have explored prompt refinement as a modular alternative, but most adopt a feed-forward approach that applies a single refined prompt throughout the entire sampling trajectory, thereby failing to fully leverage the sequential nature of reinforcement learning. To address this, here we introduce *PromptLoop*, a plug-and-play RL framework that incorporates latent feedback into step-wise prompt refinement. Rather than modifying diffusion model weights, a multimodal large language model (MLLM) is trained with RL to iteratively update prompts based on intermediate latent states of diffusion models. This design achieves a structural analogy to the Diffusion RL approach, while retaining the flexibility and generality of prompt-based alignment. Extensive experiments across diverse reward functions and diffusion backbones demonstrate that PromptLoop (i) achieves effective reward optimization, (ii) generalizes seamlessly to unseen models, (iii) composes orthogonally with existing alignment methods, and (iv) mitigates over-optimization and reward hacking.

Primary Area: generative models

Submission Number: 5170