When Bias Helps Learning: Bridging Initial Prejudice and Trainability

ICLR 2026 Conference Submission20499 Authors

Keywords: Trainability, initial guessing bias, mean field regime, phase diagrams

TL;DR: We prove theoretically that the optimal trainable state is necessarily biased in a wide range of models.

Abstract:

Understanding the statistical properties of deep neural networks (DNNs) at initialization is crucial for elucidating both their trainability and the intrinsic architectural biases they encode prior to data exposure. Mean-field (MF) analyses have demonstrated that the parameter distribution in randomly initialized networks dictates whether gradients vanish or explode. Recent work has shown that untrained DNNs exhibit an initial-guessing bias (IGB), in which large regions of the input space are assigned to a single class.
In this work, we provide a theoretical proof linking IGB to MF analyses, establishing that a network’s predisposition toward specific classes is intrinsically tied to the conditions for efficient learning. This connection leads to a counterintuitive conclusion: the initialization that optimizes trainability is systematically biased rather than neutral. We validate our theory through experiments across multiple architectures and datasets.

Primary Area: learning theory

Submission Number: 20499