Target Label-Free Confidence Calibration Under Label Shift

Jinzong Dong, Zhaohui Jiang, Tok Wang Ling

Keywords: Label Shift, Confidence Calibration, Classification

Abstract: Confidence calibration of classification models is crucial in safety-critical decision-making fields and has received extensive attention. However, general confidence calibration methods rely on the presumption that training and test data are independent and identically distributed ($i.i.d.$), which is often ineffective in real-world data where label shifts often exist. Previous works on confidence calibration under label shift heavily rely on the perception of the target domain label distribution, while the target domain's label distribution is usually unavailable in practice. To overcome this limitation, this paper explores a principled confidence calibration method under label shift that does not require any target domain label information, named Target Label-Free Confidence Calibration (TLFCC), which is realized by utilizing available variables to principledly replace variables related to the label distribution of target domain. Theoretically, this method is proven to achieve approximately correct calibration with high probability, with sample complexity comparable to histogram binning. In addition, this paper proposes a simulation data generation method for confidence calibration under label shift, which can serve as a benchmark to illustrate the discrepancy between the estimated calibration curve and the true calibration curve in the target domain, thereby reflecting the effectiveness of the calibration method. The effectiveness of our calibration method is verified in simulated and real-world data. We believe that our exploration on confidence calibration under label shift will contribute to the development of better-calibrated models, ultimately contributing to the advancement of trustworthy AI.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 2925