Diffusion Gaussian Processes

Gloria Sun, Julien Horwood, Jihao Andreas Lin, José Miguel Hernández-Lobato

Keywords: Probabilistic machine learning, Generative modelling, Bayesian inference, Diffusion models, Gaussian Processes

Abstract: We propose Diffusion Gaussian Processes (DGPs), a novel framework that integrates Gaussian processes (GPs) into Denoising Diffusion Probabilistic Models (DDPMs), leveraging DDPMs' generative flexibility while embedding GPs as denoisers to improve uncertainty quantification in regression with arbitrary likelihoods. While diffusion models demonstrate strong performance in modelling complex distributions, their uncertainty estimates are limited by point estimations of neural networks, whereas GPs provide principled uncertainty estimates but are limited by scalability and assumptions of homoscedastic Gaussian likelihoods. To enable practical and scalable implementations, we introduce two instantiations based on modern GP approximations: DDPM with sparse variational GP (D-SVGP) using inducing-point variational inference, and DDPM with GP using iterative linear system solvers like stochastic dual descent (D-SDD). We further propose a trajectory-based method to decompose predictive variance into epistemic and aleatoric components within the DGP framework. Experiments on synthetic datasets and UCI regression benchmarks show DGPs achieving comparable predictive performances to existing baselines.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 44