Asymptotics of Wide Networks from Feynman DiagramsDownload PDF

25 Sept 2019, 19:22 (modified: 11 Mar 2020, 07:33)ICLR 2020 Conference Blind SubmissionReaders: Everyone
Original Pdf: pdf
TL;DR: A general method for computing the asymptotic behavior of wide networks using Feynman diagrams
Abstract: Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals. We apply our method to study training dynamics, improving existing bounds and deriving new results on wide network evolution during stochastic gradient descent. Going beyond the strict large width limit, we present closed-form expressions for higher-order terms governing wide network training, and test these predictions empirically.
8 Replies

Loading