Demystifying ResNet

Sihan Li, Jiantao Jiao, Yanjun Han, Tsachy Weissman

Abstract: We provide a theoretical explanation for the superb performance of ResNet via the study of deep linear networks and some nonlinear variants. We show that with or without nonlinearities, by adding shortcuts that have depth two, the condition number of the Hessian of the loss function at the zero initial point is depth-invariant, which makes training very deep models no more difficult than shallow ones. Shortcuts of higher depth result in an extremely flat (high-order) stationary point initially, from which the optimization algorithm is hard to escape. The 1-shortcut, however, is essentially equivalent to no shortcuts. Extensive experiments are provided accompanying our theoretical results. We show that initializing the network to small weights with 2-shortcuts achieves significantly better results than random Gaussian (Xavier) initialization, orthogonal initialization, and shortcuts of deeper depth, from various perspectives ranging from final loss, learning dynamics and stability, to the behavior of the Hessian along the learning process.

Keywords: Deep learning, Optimization, Theory

Conflicts: mails.tsinghua.edu.cn, mail.tsinghua.edu.cn, tsinghua.edu.cn, stanford.edu, harvard.edu

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1611.01186/code)