On the Anisotropy of Score-Based Generative Models

ICLR 2026 Conference Submission12968 Authors

Keywords: Optimization, Deep Learning, Diffusion Models, Inductive Biases

TL;DR: We investigate the role of network architecture in shaping the directional inductive biases of modern score-based generative models.

Abstract: We investigate the role of network architecture in shaping the inductive biases of modern score-based generative models. To this end, we introduce the Score Anisotropy Directions (SADs), architecture-dependent directions that reveal how different networks preferentially capture data structure. Our analysis shows that SADs form adaptive bases aligned with the architecture's output geometry, providing a principled way to predict generalization ability in score models prior to training. Through both synthetic data and standard image benchmarks, we demonstrate that SADs reliably capture fine-grained model behavior and correlate with downstream performance, as measured by Wasserstein metrics. Our work offers a new lens for explaining and predicting directional biases of generative models. Code to reproduce our experiments is included in the supplementary material.

Primary Area: optimization

Submission Number: 12968