Exponentially vanishing sub-optimal local minima in multilayer neural networks

Daniel Soudry, Elad Hoffer

Keywords: neural networks, theory, optimization, local minima, loss landscape

TL;DR: "Bad" local minima are vanishing in a multilayer neural net: a proof with more reasonable assumptions than before

Abstract:

Background: Statistical mechanics results (Dauphin et al. (2014); Choromanska et al. (2015)) suggest that local minima with high error are exponentially rare in high dimensions. However, to prove low error guarantees for Multilayer Neural Networks (MNNs), previous works so far required either a heavily modified MNN model or training method, strong assumptions on the labels (e.g., “near” linear separability), or an unrealistically wide hidden layer with \Omega\(N) units.
Results: We examine a MNN with one hidden layer of piecewise linear units, a single output, and a quadratic loss. We prove that, with high probability in the limit of N\rightarrow\infty datapoints, the volume of differentiable regions of the empiric loss containing sub-optimal differentiable local minima is exponentially vanishing in comparison with the same volume of global minima, given standard normal input of dimension d_0=\tilde{\Omega}(\sqrt{N}), and a more realistic number of d_1=\tilde{\Omega}(N/d_0) hidden units. We demonstrate our results numerically: for example, 0% binary classification training error on CIFAR with only N/d_0 = 16 hidden neurons.

Code: [ MNNsMinima/Paper](https://github.com/MNNsMinima/Paper)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1702.05777/code)