Learnable Uncertainty under Laplace Approximations

Agustinus Kristiadi, Matthias Hein, Philipp Hennig

Keywords: Bayesian deep learning, Laplace approximations, uncertainty quantification

Abstract: Laplace approximations are classic, computationally lightweight means to construct Bayesian neural networks (BNNs). As in other approximate BNNs, one cannot necessarily expect the induced predictive uncertainty to be calibrated. Here we develop a formalism to explicitly "train" the uncertainty in a decoupled way to the prediction itself. To this end we introduce uncertainty units for Laplace-approximated networks: Hidden units with zero weights that can be added to any pre-trained, point-estimated network. Since these units are inactive, they do not affect the predictions. But their presence changes the geometry (in particular the Hessian) of the loss landscape around the point estimate, thereby affecting the network's uncertainty estimates under a Laplace approximation. We show that such units can be trained via an uncertainty-aware objective, making the Laplace approximation competitive with more expensive alternative uncertainty-quantification frameworks.

One-sentence Summary: A new form of hidden units for Bayesian deep learning that only affects uncertainty, not the prediction, and which can be trained post-hoc at low cost.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.02720/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=LgfpwqInwx