Towards Unbiased Calibration using Meta-Regularization

Cheng Wang, Jacek Golebiowski

Abstract: Model miscalibration has been frequently identified in modern deep neural networks. Recent work aims to improve model calibration directly through a differentiable calibration proxy. However, the calibration produced is often biased due to the binning mechanism. In this work, we propose to learn better-calibrated models via meta-regularization, which has two components: (1) gamma network (gamma-net), a meta learner that outputs sample-wise gamma value (continuous variable) for Focal loss for regularizing the backbone network; (2) smooth expected calibration error (SECE), a Gaussian-kernel based, unbiased, and differentiable surrogate to ECE that enables the smooth optimization of gamma-net. We evaluate the effectiveness of the proposed approach in regularizing neural networks towards improved and unbiased calibration on three computer vision datasets. We empirically demonstrate that: (a) learning sample-wise $\gamma$ as continuous variables can effectively improve calibration; (b) SECE smoothly optimizes gamma-net towards unbiased and robust calibration with respect to the binning schemes; and (c) the combination of gamma-net and SECE achieves the best calibration performance across various calibration metrics while retaining very competitive predictive performance as compared to multiple recently proposed methods.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We have incorporated the action editor's comment and implemented the following revisions into the camera ready version:

1. We addressed the final comment from the action editor by stressing the significance of our proposed method in the Section 5.1.

2. Restructured the Section 3 Preliminaries in a more understandable way

3. Fixed formatting/typos.

Assigned Action Editor: ~Bruno_Loureiro1

Submission Number: 2358