A visual exploration of Gaussian Processes and Infinite Neural Networks

Anonymous

Keywords: Infinite Neural Networks, Gaussian Processes, Bayesian Inference, Machine Learning

Abstract:

Gaussian Processes generalize Gaussian distributions to functons as random variables (stochastic processes). As such, they are a powerful tool for regression and Bayesian inference. In the last years, they have received increased attention due to the interest in machine learning. An interesting connection between Gaussian Processes and Neural Networks exists when the width of feed-forward neural networks tends towards infinity. For a specific choice of priors over the network parameters, the resulting network samples will be distributed according to a Gaussian Process.

In this blog post, I intend to explore this connection between Gaussian Processes and infinite Neural Nets, using Julia code and visual examples. I believe that there is some value in creating an intuitive understanding of the connection between those two methods as Gaussian Processes may help us understand how well Neural Networks can perform in the limit of infinite neurons per layer. Moreover, they may do so while keeping the probabilistic nature of the process in mind, as network training and the resulting performance will heavily depend on the initialization of the network weights. Finally, I discuss the limitations of this approach and how it may be extended by using Bayesian Neural Networks.

ICLR Paper: https://openreview.net/pdf?id=B1EA-M-0Z, https://openreview.net/forum?id=rJl-b3RcF7