Multiclass Alignment of Confidences and Softened Target Occurrences for Train-time Calibration

Vinith Kugathasan, Honglu Zhou, Zachary Izzo, Gayal Kuruppu, Sejong Yoon, Muhammad Haris Khan

Keywords: Model calibration, network calibration, calibration

TL;DR: The paper proposes a novel train-time calibration method which is based on multiclass alignment of confidences with the gradually softened target occurrences.

Abstract: In spite of delivering remarkable predictive accuracy across many domains, including computer vision and medical imaging, Deep Neural Networks (DNNs) are susceptible to making overconfident predictions. This could potentially limit their utilization and adoption in many real-world applications, especially involving security-sensitive decision making. Among existing approaches to model calibration, post-hoc based techniques are simple and effective, however, they require a separate hold-out data. Lately, train-time calibration has emerged as an alternate paradigm, in which the recent methods have shown state-of-the-art calibration results. Inspired by the train-time calibration direction, in this paper, we propose a novel train-time calibration method at the core of which is an auxiliary loss formulation, namely multiclass alignment of confidences with the gradually softened ground truth occurrences (MACSO). It is developed on the intuition that, for a class, the gradually softened ground truth occurrences distribution is a suitable non-zero entropy signal whose better alignment with the predicted confidences distribution is positively correlated with reducing the model calibration error. In our train-time approach, besides simply aligning the two distributions, e.g., via their means or KL divergence, we propose to quantify the linear correlation between the two distributions which preserves the relations among them, thereby further improving the calibration performance. Extensive results on several challenging datasets, featuring in and out-of-domain scenarios, class imbalanced problem, and a medical image classification task, validate the efficacy of our method against state-of-the-art train-time calibration methods.

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4370