Estimating Expected Calibration Error for Positive-Unlabeled Learning

TMLR Paper5463 Authors

Abstract:

The reliability of probabilistic classifiers hinges on their calibration---the property that their confidence accurately reflect the true class probabilities.
The expected calibration error (ECE) is a standard metric for quantifying the calibration of classifiers.
However, its estimation presumes access to ground-truth labels.
In positive-unlabeled (PU) learning, only positive and unlabeled data are available, which makes the standard ECE estimator inapplicable.
Although PU learning has been extensively studied for risk estimation and classifier training, calibration in this setting has received little attention.
In this paper, we present PU-ECE, the first ECE estimator for PU data.
We provide non-asymptotic bias bounds and prove convergence rates that match those of the fully supervised ECE with an optimal bin size.
Furthermore, we develop an information-theoretic generalization error analysis of PU-ECE by formalizing the conditional mutual information (CMI) for a PU setting.
Experiments on synthetic and real-world benchmark datasets validate our theoretical analysis and demonstrate that our PU-based ECE estimator achieves performance comparable to that of the fully-labeled ECE estimator.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

Addressed the comments by Reviewer z4Bc.

**Clarifications in Section 2**
- Revised the section 2, generalizing the choice of the output activation function for $f(X)$, as pointed out in the review.
- Revised the section 2.2.2, clarifying the definition of $n_\mathrm{e}$ and $B$ and addressing the ambiguity noted in the review.

**Appendix Additions**
- Added an appendix discussing the extension of PU-ECE to multiclass PU learning.
- Added an appendix discussing reliability diagrams by PU data, to provide a visual illustration of calibration and another useful application of our approach.

Assigned Action Editor: ~Bruno_Loureiro1

Submission Number: 5463