DiSA: Diffusion Step Annealing in Autoregressive Image Generation

ICLR 2026 Conference Submission15112 Authors

Keywords: Autoregressive Image Generation, Diffusion Model, Fast Sampling

TL;DR: We propose DiSA, a simple training-free method that reduces diffusion steps during autoregressive generation, achieving significantly faster inference while preserving generation quality.

Abstract:

An increasing number of autoregressive (AR) models, such as MAR, FlowAR, xAR, and Harmon adopt diffusion sampling to improve the quality of image generation. However, this strategy leads to low inference efficiency, because it usually takes 50 to 100 steps for diffusion to sample a token. This paper explores how to effectively address this issue.
Our key motivation is that as more tokens are generated during the AR process, subsequent tokens follow more constrained distributions and are easier to sample. To intuitively explain, if a model has generated part of a dog, the remaining tokens must complete the dog and thus are more constrained. Empirical evidence supports our motivation: at later generation stages, the next tokens can be well predicted by a multilayer perceptron, exhibit low variance, and follow closer-to-straight-line denoising paths from noise to tokens.
Based on our finding, we introduce diffusion step annealing (DiSA), a training-free method that gradually uses fewer diffusion steps as more tokens are generated, e.g., using 50 steps at the beginning and gradually decreasing to 5 steps at later stages. Because DiSA is derived from our finding specific to diffusion in AR models, it is complementary to existing acceleration methods designed for diffusion alone.
DiSA can be implemented in only a few lines of code on existing models, and albeit simple, achieves $5-10\times$ faster inference for MAR and Harmon and $1.4-2.5\times$ for FlowAR and xAR, while maintaining the generation quality.

Primary Area: generative models

Submission Number: 15112