Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance

ICLR 2026 Conference Submission14638 Authors

Keywords: Preference learning; Diffusion models; Text-to-image generation

TL;DR: Rethinking the role of CFG-style inference in preference learning for diffusion models.

Abstract: Aligning large-scale text-to-image diffusion models with nuanced human preferences remains a significant challenge. Direct preference optimization (DPO), while efficient and effective, often suffers from generalization gap in large-scale finetuning. We take inspiration from test-time guidance techniques and view preference alignment as a variant of classifier-free guidance (CFG), where a finetuned preference model serves as an external control signal. This perspective yields a simple and effective method that improves alignment with human preferences. To further improve generalization, we decouple preference learning into two modules trained on positive and negative samples, whose combination at inference can yield a more effective alignment signal. We quantitatively and qualitatively validate our approach on Stable Diffusion 1.5 and Stable Diffusion XL using standard image preference datasets such as Pick-a-Pic v2 and HPDv3.

Primary Area: generative models

Submission Number: 14638