Alignment of Diffusion Model and Flow Matching for Text- to-Image Generation

TMLR Paper9091 Authors

Abstract: Diffusion models and flow matching have demonstrated remarkable success in text-to-image generation. While many existing alignment methods primarily focus on fine-tuning pre-trained generative models to maximize a given reward function, these approaches require extensive computational resources and may not generalize well across different objectives. In this work, we propose a novel alignment framework by leveraging the underlying nature of the alignment problem---sampling from reward-weighted distributions---and show that it applies to both diffusion models (via score guidance) and flow matching models (via velocity guidance). We show that the score function (velocity field) required for the reward-weighted distribution can be decomposed into the pre-trained score (velocity field) plus a conditional expectation of the reward. For the alignment on the diffusion model, we identify a fundamental challenge: the adversarial nature of the guidance term can introduce undesirable artifacts in the generated images. Therefore, we propose a finetuning-free framework that trains a guidance network to estimate the conditional expectation of the reward. We achieve comparable performance to finetuning-based models with one-step generation with at least a 60\% reduction in computational cost. For the alignment on flow matching, we propose a training-free framework that improves the generation quality without additional computational cost.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Changyou_Chen1

Submission Number: 9091