Uncertainty Preservation in Generative Visual Autoregression

Mina Rezaei, Meimingwei Li, Julian Rodemann, Stefan Andreas Baumann, Bernd Bischl, Björn Ommer, Eyke Hüllermeier

Keywords: Image Generation, Second-order Uncertainty, Semantic Entropy

Abstract: Autoregressive (AR) models are among the most popular frameworks for visual generation. Recently, they have demonstrated competitive performance in visual generation through next-scale prediction. However, the training pipeline relies on cross-entropy as an objective, which enforces a precise target distribution for each token. This can lead to overconfident, sharp predictions and ultimately, a lack of diversity or mode collapse in the generated samples. In supervised learning, techniques such as label relaxation have been proposed to address the shortcomings of cross-entropy by replacing precise targets with sets of plausible distributions. However, in generative models, uncertainty should be considered as dispersion over a range of plausible outcomes rather than ambiguity about a single correct label. Building on this view, we introduce *uncertainty preservation* for visual autoregressive models. Specifically, we obtain predictive distributions as *second-order likelihoods* and penalize any deviations in their dispersion from a calibrated reference. Moreover, we implement a semantic entropy loss to provide a complementary measure of uncertainty that aligns with consistency at the meaning level. In theory, we demonstrate that our approach is a special case of second-order regularization, whereby penalizing variance deviation is equivalent to controlling the scale component of a Wasserstein distance between credal distributions. Through extensive experiments on multiple settings and datasets, we demonstrate that our model, despite its simplicity and low computational overhead, improves generation quality and diversity. Moreover, we highlight the practical utility of uncertainty quantification in image generation by showcasing additional applications where our approach could be beneficial.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 2839