Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape

Published: 04 Jun 2024, Last Modified: 04 Jun 2024Accepted by TMLREveryoneRevisionsBibTeX
Abstract: We study the loss landscape of both shallow and deep, mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank Jacobian to many regions having deficient rank depending on the amount of overparameterization.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Atsushi_Nitanda1
Submission Number: 2147