Exact Gaussian Processes on a Million Data Points

Ke Alexander Wang, Geoff Pleiss, Jacob R Gardner, Stephen Tyree, Kilian Weinberger, Andrew Gordon Wilson

Abstract: Gaussian processes (GPs) are flexible non-parametric models whose representational power grows with the size of the training data. In this paper, we develop a methodology to scale exact GPs to over 10^6 training points, a task previously thought to be impossible with current computing hardware. Our approach builds upon the conjugate-gradient-based inference of Gardner et al., 2018, which by itself scales to roughly 10^4 training points. With multi-GPU parallelization and more powerful preconditioning, we are able to scale exact GP inference to datasets two orders of magnitude larger than previous work. A million points requires three days on $8$ GPUs for hyperparameter training and computing prediction caches, but at test time we can compute all predictive means and variances in under a second using only one GPU. Leveraging these capabilities, we perform the first-ever comparison of exact GPs against scalable GP approximations on datasets with 10^4-10^6 data points, showing dramatic performance improvements.

Code Link: https://github.com/cornellius-gp/gpytorch/blob/master/examples/01_Simple_GP_Regression/Simple_MultiGPU_GP_Regression.ipynb

CMT Num: 8279