Normalizing Flows for Calibration and Recalibration

Achintya Gopal, Aaron Key

Keywords: recalibration, normalizing flows, uncertainty

Abstract:

In machine learning, due to model misspecification and overfitting, estimates of the aleatoric uncertainty are often inaccurate.
One approach to fix this is isotonic regression, in which a monotonic function is fit on a validation set to map the model's CDF to an optimally calibrated CDF. However, this makes it infeasible to compute additional statistics of interest on the model distribution (such as the mean). In this paper, through a reframing of recalibration as MLE, we replace isotonic regression with normalizing flows. This allows us to retain the ability to compute the statistical properties of the model (such as closed-form likelihoods, mean, correlation, etc.) and provides an opportunity for additional capacity at the cost of possible overfitting. Most importantly, the fundamental properties of normalizing flows allow us to generalize recalibration to conditional and multivariate distributions. To aid in detecting miscalibration and measuring our success at fixing it, we use a simple extension of the calibration Q-Q plot.

One-sentence Summary: This paper introduces using normalizing flows for recalibration and extends recalibration to the multivariate case.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=5TXLG56Pj