Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error

Ondrej Bohdal, Yongxin Yang, Timothy Hospedales

Abstract: Calibration of neural networks is a topical problem that is becoming more and more important as neural networks increasingly underpin real-world applications. The problem is especially noticeable when using modern neural networks, for which there is a significant difference between the confidence of the model and the probability of correct prediction. Various strategies have been proposed to improve calibration, yet accurate calibration remains challenging. We propose a novel framework with two contributions: introducing a new differentiable surrogate for expected calibration error (DECE) that allows calibration quality to be directly optimised, and a meta-learning framework that uses DECE to optimise for validation set calibration with respect to model hyper-parameters. The results show that we achieve competitive performance with existing calibration approaches. Our framework opens up a new avenue and toolset for tackling calibration, which we believe will inspire further work on this important challenge.

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/ondrejbohdal/meta-calibration

Assigned Action Editor: ~Yingzhen_Li1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1189