Leveraging Label Dependencies for Calibration in Multi-Label Classification through Proper Scoring Rule

ICLR 2026 Conference Submission23753 Authors

Keywords: Calibration, Strictly Proper Loss, Multi-Label Classification, Label Dependency, CMLL Loss

Abstract: Modern Deep Neural Networks (DNNs) trained by using cross entropy for binary or multi-class classification are known to produce poorly calibrated probability estimates. While various calibration methods have been proposed, only a few addresses the challenge of calibrating Multi-Label Classification (MLC) tasks. Multi-label classification is essential in real-world applications, as most objects or instances naturally belong to multiple categories, and the associated labels often exhibit strong interdependencies. A key difficulty in calibrating MLC models lies in effectively considering the information of label interdependencies. Existing methods that attempt to model the label interdependencies often lack rigorous statistical justification or they consider the labels are independent or lacks being strictly proper - a property which induces calibrated predicted probabilities upon minimization. In this work, we introduce a novel loss function, \emph{Correlated Multi-Label Loss (CMLL)}, that explicitly captures label interdependencies while satisfying the properties of a strictly proper loss. Our method leverages pairwise label correlations to incorporate dependency information into the training process and is proven to be Fisher consistent. Extensive experiments on three publicly available benchmark multi-label datasets demonstrate the effectiveness of our approach. Our proposed method significantly reduces calibration error while maintaining state-of-the-art classification accuracy.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 23753