Deep neural networks have an inbuilt Occam’s razor

Chris Mingard, Henry Rees, Guillermo Valle-Pérez, Ard A. Louis

Abstract: The remarkable performance of overparameterized deep neural networks (DNNs) must arise from an interplay between network architecture, training algorithms, and structure in the data. To disentangle these three components for supervised learning, we apply a Bayesian picture based on the functions expressed by a DNN. The prior over functions is determined by the network architecture, which we vary by exploiting a transition between ordered and chaotic regimes. For Boolean function classification, we approximate the likelihood using the error spectrum of functions on data. Combining this with the prior yields an accurate prediction for the posterior, measured for DNNs trained with stochastic gradient descent. This analysis shows that structured data, together with a specific Occam’s razor-like inductive bias towards (Kolmogorov) simple functions that exactly counteracts the exponential growth of the number of functions with complexity, is a key to the success of DNNs. Deep neural networks predict well despite having many more parameters than data points. The authors show that deep neural networks prefer simpler solutions due to an inbuilt “Occam’s razor”. Favouring simplicity avoids overfitting and captures patterns in data, explaining their success.

External IDs:doi:10.1038/s41467-024-54813-x