Conformal Prediction for Deep Classifier via Label Ranking

Jianguo Huang, HuaJun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, Hongxin Wei

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: classification, conformal prediction, uncertainty estimation

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A novel conformal prediction algorithm based on label ranking quantifies the prediction uncertainty of multi-classes classifiers.

Abstract:

Conformal prediction is a statistical framework that generates prediction sets containing ground-truth labels with a desired coverage guarantee.
The predicted probabilities produced by machine learning models are generally miscalibrated, leading to large prediction sets in conformal prediction.
In this paper, we empirically and theoretically show that disregarding the probabilities' value will mitigate the undesirable effect of miscalibrated probability values.
Then, we propose a novel algorithm named $\textit{Sorted Adaptive prediction sets}$ (SAPS), which discards all the probability values except for the maximum softmax probability.
The key idea behind SAPS is to minimize the dependence of the non-conformity score on the probability values while retaining the uncertainty information.
In this manner, SAPS can produce sets of small size and communicate instance-wise uncertainty.
Theoretically, we provide a finite-sample coverage guarantee of SAPS and show that the expected value of set size from SAPS is always smaller than APS.
Extensive experiments validate that SAPS not only lessens the prediction sets but also broadly enhances the conditional coverage rate and adaptation of prediction sets.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6778