Assuming Locally Equal Calibration Errors for Non-Parametric Multiclass Calibration

Kaspar Valk, Meelis Kull

Abstract: A probabilistic classifier is considered calibrated if it outputs probabilities equal to the expected class distribution given the classifier's output. Calibration is essential in safety-critical tasks where small deviations between the predicted probabilities and the actually observed class proportions can incur high costs. A common approach to improve the calibration of a classifier is to use a hold-out data set and a post-hoc calibration method to learn a correcting transformation for the classifier's output. This work explores the field of post-hoc calibration methods for multi-class classifiers and formulates two assumptions about the probability simplex which have been used by many existing non-parametric calibration methods, but despite this, have never been explicitly stated: assuming locally equal label distributions or assuming locally equal calibration errors. Based on the latter assumption, an intuitive non-parametric post-hoc calibration method is proposed, which is shown to offer improvements to the state-of-the-art according to the expected calibration error metric on CIFAR-10 and CIFAR-100 data sets.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

We have moved to the appendix:
* the toy example of different calibration definitions from Section 2.2,
* the discussion on computational and memory complexity of LECE calibration from Section 3.2,
* the discussion on running times of LECE calibration from Section 4.2.

We have also removed the colors which denoted the changes done addressing the reviewers' feedback.

We have done minor edits to improve the layout and locations of the tables and figures in the paper.

We have deanonymized the paper, added an acknowledgements section, and added an URL to the source code of the experiments to the end of the introduction.

Code: https://github.com/kaspar98/lece-calibration

Assigned Action Editor: ~Aditya_Menon1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 879