Implicit Regularization Through Hidden Diversity in Neural Networks

ICLR 2026 Conference Submission24472 Authors

Keywords: implicit regularization, implicit ensembles, neural networks, bias-variance tradeoff, double descent, generalization

TL;DR: By interpreting the neural network as an implicit ensemble, we expose an additional term in the bias-variance decomposition called diversity, which acts as an implicit regularizer.

Abstract: A significant body of work has focused on studying the mechanisms behind the implicit regularization in neural networks. Recently, developments in ensemble theory have demonstrated that, for a wide variety of loss functions, the expected risk of the ensemble can be decomposed into a bias and variance term together with an additional term called *diversity*. By using this theoretical framework and by interpreting a *single* neural network as an ensemble, we expose a hidden diversity term in the decomposition of a neural network's expected risk. We argue that the additional diversity term regulates the variance error, thus identifying a new source of *implicit regularization* in neural networks. We demonstrate this regularization on regression and classification datasets by estimating the bias, variance, and diversity terms for both MLPs and CNNs. Using double descent as an example, we observe that diversity significantly increases for wide overparameterized neural networks. These results demonstrate a new perspective on implicit regularization in neural networks and open new possible avenues of research into their generalization.

Primary Area: learning theory

Submission Number: 24472