The Multilinear Structure of ReLU Networks

Thomas Laurent, James von Brecht

Abstract: We study the loss surface of neural networks equipped with a hinge loss criterion and ReLU or leaky ReLU nonlinearities. Any such network defines a piecewise multilinear form in parameter space. By...