Quantile Regularization : Towards Implicit Calibration of Regression Models

Saiteja Utpala, Piyush Rai

Keywords: Calibration, Reliable Uncertainty Quantification, Probabilistic Deep Learning

Abstract:

Recent works have shown that most deep learning models are often poorly calibrated, i.e., they may produce overconfident
predictions that are wrong, implying that their uncertainty estimates are unreliable. While a number of approaches have been proposed recently to calibrate classification models, relatively little work exists on calibrating regression models. Isotonic Regression has recently been advocated for regression calibration. We provide a detailed formal analysis of the \emph{side-effects} of Isotonic Regression when used for regression calibration. To address this, we recast quantile calibration as entropy estimation, and leverage this idea to construct a novel quantile regularizer, which can be used in any optimization based probabilisitc regression models. Unlike most of the existing approaches for calibrating regression models, which are based on \emph{post-hoc} processing of the model's output, and require an additional dataset, our method is trainable in an end-to-end fashion, without requiring an additional dataset. We provide empirical results demonstrating that our approach improves calibration for regression models trained on diverse architectures that provide uncertainty estimates, such as Dropout VI, Deep Ensembles

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: an implicit calibration approach for regression models

Reviewed Version (pdf): https://openreview.net/references/pdf?id=Fztm6VTC6L