Deep Ensemble as a Gaussian Process Posterior

Zhijie Deng, Feng Zhou, Jianfei Chen, Guoqiang Wu, Jun Zhu

Keywords: Deep ensemble, Bayesian deep learning, Gaussian process, functional variational inference, uncertainty estimation

Abstract: Deep Ensemble (DE) is a flexible, feasible, and effective alternative to Bayesian neural networks (BNNs) for uncertainty estimation in deep learning. However, DE is broadly criticized for lacking a proper Bayesian justification. Some attempts try to fix this issue, while they are typically coupled with a regression likelihood or rely on restrictive assumptions. In this work, we propose to define a Gaussian process (GP) approximate posterior with the ensemble members, based on which we perform variational inference directly in the function space. We further develop a function-space posterior regularization mechanism to properly incorporate prior knowledge. We demonstrate the algorithmic benefits of variational inference in the GP family, and provide strategies to make the training feasible. As a result, our method consumes only marginally added training cost than the standard Deep Ensemble. Empirically, our approach achieves better uncertainty estimation than the existing Deep Ensemble and its variants across diverse scenarios.

One-sentence Summary: We suggest that deep ensemble defines a Gaussian process (GP) approximate posterior, based on which we develop a principled Bayesian learning rule for deep ensemble.