Optimizer Choice Matters For The Emergence of Neural Collapse

ICLR 2026 Conference Submission18037 Authors

Keywords: neural collapse, implicit bias, deep learning theory, classification, adaptive optimizers, training dynamics, adamw, sgd, weight decay

TL;DR: We show theoretically and empirically that the optimizer choice affects the emergence of neural collapse (NC). In particular, we show that the dynamics of NC metrics differ qualitatively depending whether coupled or decoupled weight decay is used.

Abstract:

Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the optimizer, thereby suggesting that NC is universal across optimization methods.
In this work, we challenge this assumption and demonstrate that the choice of optimizer plays a critical role in the emergence of NC. The phenomenon is typically quantified through NC metrics, which, however, are difficult to track and analyze theoretically. To overcome this limitation, we introduce a novel diagnostic metric, NC0, whose convergence to zero is a necessary condition for NC. Using NC0, we provide theoretical evidence that NC cannot emerge under decoupled weight decay, as implemented in AdamW. Concretely, we prove that SGD, SignGD with coupled weight decay (a special case of Adam), and SignGD with decoupled weight decay (a special case of AdamW) exhibit qualitatively different NC0 dynamics.
Finally, we conduct extensive empirical experiments consisting of 3,900 training runs across various datasets, architectures, optimizers, and hyperparameters,
confirming our theoretical results.
This work provides the first theoretical explanation for optimizer-dependent emergence of NC and highlights
the overlooked role of weight decay coupling in implicit biases of optimizers.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 18037