Deep Convolutional Networks as shallow Gaussian ProcessesDownload PDF

27 Sept 2018, 22:37 (modified: 10 Feb 2022, 11:31)ICLR 2019 Conference Blind SubmissionReaders: Everyone
Keywords: Gaussian process, CNN, ResNet, Bayesian
TL;DR: We show that CNNs and ResNets with appropriate priors on the parameters are Gaussian processes in the limit of infinitely many convolutional filters.
Abstract: We show that the output of a (residual) CNN with an appropriate prior over the weights and biases is a GP in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GP with a comparable number of parameters.
Code: [![github](/images/github_icon.svg) rhaps0dy/convnets-as-gps](https://github.com/rhaps0dy/convnets-as-gps) + [![Papers with Code](/images/pwc_icon.svg) 2 community implementations](https://paperswithcode.com/paper/?openreview=Bklfsi0cKm)
7 Replies

Loading