The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks

Itay Safran, Gilad Yehudai, Ohad Shamir

Abstract: We study the effects of mild over-parameterization on the optimization landscape of a simple ReLU neural network of the form $\mathbf{x}\mapsto\sum_{i=1}^k\max\{0,\mathbf{w}_i^{\top}\mathbf{x}\}$, ...