Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks

ICLR 2026 Conference Submission18886 Authors

Keywords: Neural scaling laws, Implicit bias, Learning curves, Spectral complexity norm, Perceptron theory

TL;DR: We connect neural scaling laws in deep networks with the implicit bias induced by logistic losses through a surprisingly simple perceptron theory.

Abstract: Scaling laws in deep learning -- empirical power-law relationships linking model performance to resource growth -- have emerged as simple yet striking regularities across architectures, datasets, and tasks. These laws are particularly impactful in guiding the design of state-of-the-art models, since they quantify the benefits of increasing data or model size, and hint at the foundations of interpretability in machine learning. However, most studies focus on asymptotic behavior at the end of training. In this work, we describe a richer picture by analyzing the entire training dynamics: we identify two novel \textit{dynamical} scaling laws that govern how performance evolves as function of different norm-based complexity measures. Combined, our new laws recover the well-known scaling for test error at convergence. Our findings are consistent across CNNs, ResNets, and Vision Transformers trained on MNIST, CIFAR-10 and CIFAR-100. Furthermore, we provide analytical support using a single-layer perceptron trained with logistic loss, where we derive the new dynamical scaling laws, and we explain them through the implicit bias induced by gradient-based training.

Primary Area: optimization

Submission Number: 18886