A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate CaseDownload PDF

25 Sept 2019, 19:30 (modified: 11 Mar 2020, 07:33)ICLR 2020 Conference Blind SubmissionReaders: Everyone
Original Pdf: pdf
TL;DR: We characterize the space of functions realizable as a ReLU network with an unbounded number of units (infinite width), but where the Euclidean norm of the weights is bounded.
Abstract: We give a tight characterization of the (vectorized Euclidean) norm of weights required to realize a function $f:\mathbb{R}\rightarrow \mathbb{R}^d$ as a single hidden-layer ReLU network with an unbounded number of units (infinite width), extending the univariate characterization of Savarese et al. (2019) to the multivariate case.
Keywords: inductive bias, regularization, infinite-width networks, ReLU networks
7 Replies

Loading