Approximate Inference for Fully Bayesian Gaussian Process Regression

Vidhi Lalchand, Carl Edward Rasmussen

Keywords: Gaussian Processes, Hierarchical Models, MCMC, Variational Inference

TL;DR: Analysis of Bayesian Hyperparameter Inference in Gaussian Process Regression

Abstract: Learning in Gaussian Process models occurs through the adaptation of hyperparameters of the mean and the covariance function. The classical approach entails maximizing the marginal likelihood yielding fixed point estimates (an approach called Type II maximum likelihood or ML-II). An alternative learning procedure is to infer the posterior over hyperparameters in a hierarchical specification of GPs we call Fully Bayesian Gaussian Process Regression (GPR). This work considers two approximations to the intractable hyperparameter posterior, 1) Hamiltonian Monte Carlo (HMC) yielding a sampling based approximation and 2) Variational Inference (VI) where the posterior over hyperparameters is approximated by a factorized Gaussian (mean-field) or a full rank Gaussian accounting for correlations between hyperparameters. We analyse the predictive performance for fully Bayesian GPR on a range of benchmark data sets.