Nonparametric Distribution Regression Re-calibration

Ádám Jung, Domokos M. Kelen, Andras A Benczur

Abstract: A key challenge in probabilistic regression is ensuring that predictive distributions accurately reflect true empirical uncertainty. Minimizing overall prediction error often encourages models to prioritize informativeness over calibration, producing narrow but overconfident predictions. However, in safety-critical settings, trustworthy uncertainty estimates are often more valuable than narrow intervals. Realizing the problem, several recent works have focused on post-hoc corrections; however, existing methods either rely on weak notions of calibration (such as PIT uniformity) or impose restrictive parametric assumptions on the nature of the error. To address these limitations, we propose a novel nonparametric re-calibration algorithm based on conditional kernel mean embeddings, capable of correcting calibration error without restrictive modeling assumptions. For efficient inference with real-valued targets, we introduce a novel characteristic kernel over distributions that can be evaluated in $\mathcal{O}(n \log n)$ time for empirical distributions of size $n$. We demonstrate that our method consistently outperforms prior re-calibration approaches across a diverse set of regression benchmarks and model classes.

Lay Summary:

When machine learning models make predictions, they also need to tell us how certain they are. Often, these models are trained to be as precise as possible, which can encourage overconfidence. In high-stakes situations—such as autonomous driving or healthcare—an overconfident model is dangerous; trustworthy estimates of uncertainty are far more valuable than narrow, incorrect predictions.

Researchers have tried to fix this "calibration" issue after the models are already trained. While reliable solutions exist for models that sort data into simple categories (classification), fixing models that predict continuous, real-world numbers (regression)—like forecasting exact temperatures—is much harder. Existing fixes for regression either use weak evaluation tests that hide dangerous errors (by letting overconfident and underconfident mistakes cancel each other out) or rely on strict assumptions about the exact shape of the model's errors.

To address these limitations, we created a new algorithm that fine-tunes a model's uncertainty without forcing the data to fit a predefined mathematical shape. Think of it as a flexible method that smoothly adjusts the model's confidence to match the complex reality of continuous data. We also developed a fast computational shortcut to ensure this runs quickly and efficiently on real-world datasets.

Primary Area: Probabilistic Methods->Everything Else

Keywords: Calibration, CKME, Auto-calibration, strong-calibration, distribution regression

Submission Number: 34428