Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition

Sebastian Gregor Gruber, Florian Buettner

Abstract:

Reliably estimating the uncertainty of a prediction throughout the model lifecycle is crucial in
many safety-critical applications. The most common way to measure this uncertainty is via the
predicted confidence. While this tends to work
well for in-domain samples, these estimates are
unreliable under domain drift and restricted to
classification. Alternatively, proper scores can be
used for most predictive tasks but a bias-variance
decomposition for model uncertainty does not
exist in the current literature. In this work we introduce a general bias-variance decomposition for
strictly proper scores, giving rise to the Bregman
Information as the variance term. We discover
how exponential families and the classification
log-likelihood are special cases and provide novel
formulations. Surprisingly, we can express the
classification case purely in the logit space. We
showcase the practical relevance of this decomposition on several downstream tasks, including
model ensembles and confidence regions. Further,
we demonstrate how different approximations of
the instance-level Bregman Information allow out-of-distribution detection for all degrees of domain
drift.