Next-Scale Autoregressive Models are Zero-Shot Single-Image Object View Synthesizers

Shiran Yuan, Hao Zhao

Keywords: novel view synthesis, generative modeling, autoregressive modeling

Abstract: Learning to synthesize novel views without explicit 3D representations or hand-crafted 3D inductive bias has recently gained attention: it is simpler, more formally direct, and better aligned with the lesson that scalable learning paradigms with less assumptions built into architectural design (e.g., regarding geometry) often win. However, the current dominant solutions are diffusion-based, which typically suffer from problems like slow inference. We introduce ArchonView, the first autoregressive model for zero-shot single-image, object-centric novel view synthesis (NVS), achieving substantially faster inference, higher accuracies, and notably not relying on fine-tuning of 2D generative checkpoints (challenging the common assumption that 2D priors are required in diffusion-based NVS). We design innovative methods of both global and local conditioning to suit characteristics of the NVS task. Crucially, a naïve application of next-scale autoregression fails; we identify two design choices that unlock performance: local conditioning pre-filling, and removing global AdaLN at the classifier head. ArchonView delivers state-of-the-art zero-shot results across six standard benchmarks (GSO, ABO, OmniObject3D, RTMV, NeRF-Synthetic, ShapeNet), while being several times faster than diffusion baselines (e.g., 0.22s v.s. 1.7–1.8s per view at matched parameter count). It consistently improves synthesis accuracy, and scales predictably with both model size (135M–2B) and data size, exhibiting clear scaling-law-like trends. Our findings suggest a paradigm shift and challenge an existing assumption: first, for object-centric NVS, next-scale autoregression can be faster, simpler, and more accurate than diffusion; and second, priors obtained from fine-tuning 2D-pretrained models may not be necessary for generative NVS. Our code is open-sourced at https://anonymous.4open.science/r/ArchonView/.

Primary Area: generative models

Submission Number: 3071