Better Calibration Error Estimation for Reliable Uncertainty Quantification
Keywords: uncertainty quantification, calibration error estimation
TL;DR: We propose a new and more accurate calibration error estimator that leads to more reliable uncertainty quantification.
Abstract: Reliable uncertainty quantification is crucial in high-stakes applications, such as healthcare. The $\text{ECE}_{EW}$ has been the most commonly used estimator to quantify the calibration error (CE), but it is heavily biased and can significantly underestimate the true calibration error. While alternative estimators, such as $\text{ECE}_\text{DEBIASED}$ and $\text{ECE}_\text{SWEEP}$, achieve smaller estimation bias in comparison, they exhibit a trade-off between overestimation of the CE on uncalibrated models and underestimation on recalibrated models. To address this trade-off, we propose a new estimator based on K-Nearest Neighbors (KNN), called $\text{ECE}_\text{KNN}$, which constructs representative overlapping local neighbourhoods for improved CE estimation. Empirical evaluation results demonstrate that $\text{ECE}_\text{KNN}$ simultaneously achieves near-zero underestimation of the CE on uncalibrated models while also achieving lower degrees of overestimation on recalibrated models. The implementation of our proposed $\text{ECE}_\text{KNN}$ is available at https://github.com/esterlab/KNN-ECE/.
Submission Number: 44
Loading