Fixing Asymptotic Uncertainty of Bayesian Neural Networks with Infinite ReLU Features

Agustinus Kristiadi, Matthias Hein, Philipp Hennig

Keywords: Bayesian deep learning, Gaussian processes, uncertainty quantification

Abstract: Approximate Bayesian methods can mitigate overconfidence in ReLU networks. However, far away from the training data, even Bayesian neural networks (BNNs) can still underestimate uncertainty and thus be overconfident. We suggest to fix this by considering an infinite number of ReLU features over the input domain that are never part of the training process and thus remain at prior values. Perhaps surprisingly, we show that this model leads to a tractable Gaussian process (GP) term that can be added to a pre-trained BNN's posterior at test time with negligible cost overhead. The BNN then yields structured uncertainty in the proximity of training data, while the GP prior calibrates uncertainty far away from them. As a key contribution, we prove that the added uncertainty yields cubic predictive variance growth, and thus the ideal uniform (maximum entropy) confidence in multi-class classification far from the training data.

One-sentence Summary: A way to add infinitely many ReLU features away from the support of the data to achieve uniform asymptotic uncertainty, at negligible additional cost.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=aN6F2eY1wI