Predictive Uncertainties Based on Proper Scoring Rules
Keywords: Uncertainty Quantification, Probabilistic Methods, Bayesian Methods
TL;DR: A framework for uncertainty quantification, which is 1) based on statistical notion of risk and 2) distinguish between different sources of uncertainty.
Abstract: This paper presents a theoretical framework for understanding uncertainty through the lens of statistical risks. It introduces a method to differentiate between aleatoric uncertainty, which is related to inherent data variability, and epistemic uncertainty, which is linked to lacking of best model parameters knowledge. We explain how pointwise risk can be decomposed into Bayes risk and Excess risk, showing that Excess risk, linked to epistemic uncertainty, corresponds to Bregman divergences.
To convert these theoretical risk measures into practical uncertainty estimates, we propose using a Bayesian approach, approximating the risks through posterior distributions. We validate our method on image datasets, assessing its capability to identify out-of-distribution and misclassified data using the AUROC metric. Our findings demonstrate the efficacy of this approach and provide practical insights for estimating uncertainty in real-world scenarios.
Submission Number: 81
Loading