Exact Solutions of a Deep Linear NetworkDownload PDF

Published: 31 Oct 2022, 18:00, Last Modified: 12 Oct 2022, 16:57NeurIPS 2022 AcceptReaders: Everyone
Keywords: Deep Linear Network, Exact Solution, Collapse
TL;DR: We find the analytical expression of the global minima of a deep feedforward linear network.
Abstract: This work finds the analytical expression of the global minima of a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks. Our result implies that zero is a special point in deep neural network architecture. We show that weight decay strongly interacts with the model architecture and can create bad minima at zero in a network with more than $1$ hidden layer, qualitatively different from a network with only $1$ hidden layer. Practically, our result implies that common deep learning initialization methods are insufficient to ease the optimization of neural networks in general.
Supplementary Material: pdf
14 Replies

Loading