IRIS: Intrinsic Reward Image Synthesis

ICLR 2026 Conference Submission14485 Authors

Keywords: vision-language models, reinfocement learning

Abstract:

Despite the success of Reinforcement Learning from Human Feedback (RLHF) in language reasoning, its application to autoregressive Text-to-Image (T2I) generation is often constrained by the limited availability of human preference data. This paper explores how an autoregressive T2I model can learn from internal signals without relying on external rewards or labeled data. Contrary to recent findings in text generation, we show that maximizing self-uncertainty, rather than self-certainty, improves image generation. We observe that this is because autoregressive T2I models with low uncertainty tend to generate simple and uniform images, which are less aligned with human preferences. Based on these observations, we propose **IRIS** (**I**ntrinsic **R**eward **I**mage **S**ynthesis),
the first framework to improve autoregressive T2I models with reinforcement learning using only an intrinsic reward. Empirical results demonstrate that applying IRIS to autoregressive T2I models achieves performance that is competitive with or superior to external rewards.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 14485