What Drives Compositional Generalization? The Importance of Continuous Training Objectives in Visual Generative Models

Karim Farid, Rajat Sahay, Yumna Alnaggar, Simon Schrodi, Volker Fischer, Cordelia Schmid, Thomas Brox

Keywords: compositional generalization, diffusion models, training objective, masked generative models, limitations, world model evaluation, understanding

Abstract: Compositional generalization, the ability to generate novel combinations of known concepts, is a key ingredient for visual generative models. Yet, not all mechanisms that enable or inhibit it are fully understood. In this work, we conduct a systematic study of which design choices critically determine compositional generalization in image and video generation. By isolating independent design axes, we identify two key factors that determine compositional success: (i) whether the training objective operates on a discrete or continuous distribution, and (ii) the completeness of conditioning information about constituent factors during training. We also show that relaxing the discrete loss with an auxiliary continuous latent objective can partially recover compositional performance in discrete models like MaskGIT. Our findings, corroborated by diverse compositional tasks and preliminary evidence in world models and LLMs, motivate a shift toward continuous objectives for compositional generalization.

Submission Number: 52