Width Provably Matters in Optimization for Deep Linear Neural Networks

Simon S. Du, Wei Hu

Abstract: We prove that for an $L$-layer fully-connected linear neural network, if the width of every hidden layer is $\widetilde{\Omega}\left(L \cdot r \cdot d_{out} \cdot \kappa^3 \right)$, where $r$ and $...