Single Model Uncertainty Estimation via Stochastic Data Centering

Jayaraman J. Thiagarajan, Rushil Anirudh, Vivek Narayanaswamy, Peer-timo Bremer

Keywords: uncertainty quantification, calibration, active learning, deep ensembles, sequential optimization

TL;DR: We find that training multiple NNs (with fixed init) by shifting the data (train and val) with a constant bias produces models whose inconsistencies are a strong indicator of epistemic uncertainties. We approximate this behavior with a single NN.

Abstract: We are interested in estimating the uncertainties of deep neural networks, which play an important role in many scientific and engineering problems. In this paper, we present a striking new finding that an ensemble of neural networks with the same weight initialization, trained on datasets that are shifted by a constant bias gives rise to slightly inconsistent trained models, where the differences in predictions are a strong indicator of epistemic uncertainties. Using the neural tangent kernel (NTK), we demonstrate that this phenomena occurs in part because the NTK is not shift-invariant. Since this is achieved via a trivial input transformation, we show that this behavior can therefore be approximated by training a single neural network -- using a technique that we call $\Delta-$UQ -- that estimates uncertainty around prediction by marginalizing out the effect of the biases during inference. We show that $\Delta-$UQ's uncertainty estimates are superior to many of the current methods on a variety of benchmarks-- outlier rejection, calibration under distribution shift, and sequential design optimization of black box functions. Code for $\Delta-$UQ can be accessed at github.com/LLNL/DeltaUQ

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2207.07235/code)