Fast Predictive Uncertainty for Classification with Bayesian Deep Networks

Marius Hobbhahn, Agustinus Kristiadi, Philipp Hennig

Keywords: Bayesian Deep Learning, Approximate Inference

Abstract: In Bayesian Deep Learning, distributions over the output of classification neural networks are approximated by first constructing a Gaussian distribution over the weights, then sampling from it to receive a distribution over the categorical output distribution. This is costly. We reconsider old work to construct a Dirichlet approximation of this output distribution, which yields an analytic map between Gaussian distributions in logit space and Dirichlet distributions (the conjugate prior to the categorical) in the output space. We argue that the resulting Dirichlet distribution has theoretical and practical advantages, in particular, more efficient computation of the uncertainty estimate, scaling to large datasets and networks like ImageNet and DenseNet. We demonstrate the use of this Dirichlet approximation by using it to construct a lightweight uncertainty-aware output ranking for the ImageNet setup.

One-sentence Summary: We re-use an old method (the Laplace Bridge) in the context of Bayesian Deep Learning to improve computation time of the posterior predictive significantly for all Networks that have a Gaussian over the logits.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=96GnAF_Jz