Towards Unbiased Calibration using Meta-Regularization

TMLR Paper2358 Authors

Abstract: Model miscalibration has been frequently identified in modern deep neural networks. Recent work aims to improve model calibration directly through a differentiable calibration proxy. However, the calibration produced is often biased due to the binning mechanism. In this work, we propose to learn better-calibrated models via meta-regularization which has two components: (1) gamma network (gamma-net), a meta learner to output sample-wise gamma value (continuous variable) for focal loss for regularizing the backbone network; (2) smooth expected calibration error (SECE), a Gaussian-kernel-based unbiased and differentiable surrogate to SECE that enables the smooth optimization of gamma-net. We evaluate the effectiveness of the proposed approach in regularizing neural networks towards better and unbiased calibration on three computer vision datasets. We empirically demonstrate that: (a) learning sample-wise $\gamma$ as continuous variables can effectively improve calibration; (b) SECE smoothly optimizes gamma-net towards unbiasedness and robustness to binning schemes; and (c) the combination of gamma-net and SECE achieves the best calibration performance across various calibration metrics and retains very competitive predictive performance as compared to multiple recently proposed methods.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=Yf8iHCfG4W&noteId=yeR0RxVM9s

Changes Since Last Submission:

We have incorporated the reviewers' comments and implemented the following revisions:

1. We have revised the content to address unclear points, expanded the related work section, and rectified any typos present.

2. Additional analysis and discussion concerning the trade-off between predictive performance and calibration have been included in Appendix A.4 (refer to Figure 10 and Table 5).

3. Furthermore, we have augmented the comparisons with state-of-the-art (SOTA) methods in Table 3 within the main content.

Assigned Action Editor: ~Bruno_Loureiro1

Submission Number: 2358