Function-Space Variational Inference for Deep Bayesian Classification

Jihao Andreas Lin, Joe Watson, Pascal Klink, Jan Peters

Keywords: Bayesian deep learning, image classification, functional variational inference, Dirichlet distribution

Abstract: Bayesian deep learning approaches assume model parameters to be latent random variables and infer posterior predictive distributions to quantify uncertainty, increase safety and trust, and prevent overconfident and unpredictable behavior. However, weight-space priors are model-specific, can be difficult to interpret and hard to choose. Instead of weight-space priors, we leverage function-space variational inference to apply a Dirichlet predictive prior in function space, resulting in a variational Dirichlet posterior which facilitates easier specification of epistemic uncertainty. This is achieved through the perspective of stochastic neural network classifiers as variational implicit processes, which can be trained using function-space variational inference by devising a novel Dirichlet KL estimator. Experiments on small- and large-scale image classification tasks demonstrate that our function-space inference scales to large-scale tasks and models, improves adversarial robustness and boosts uncertainty quantification across models, without influencing the in-distribution performances, architecture or model size.

One-sentence Summary: We apply function-space variational inference to Bayesian deep learning in classification tasks by assuming Dirichlet predictives for variational implicit processes.