Conformal Prediction of Classifiers with Many Classes based on Noisy Labels

Coby Penso, Jacob Goldberger, Ethan Fetaya

Abstract: Conformal Prediction (CP) is a method to control prediction uncertainty by producing a small prediction set, ensuring a predetermined probability that the true class lies within this set. This is commonly done by defining a score, based on the model predictions, and setting a threshold on this score using a validation set. In this study, we address the problem of CP calibration when we only have access to a validation set with noisy labels. We show how we can estimate the noise-free conformal threshold based on the noisy labeled data. We derive a finite-sample coverage guarantee under uniform noise that remains effective even in classification tasks with a large number of classes. We dub our approach Noise-Aware Conformal Prediction (NACP). We illustrate the performance of the proposed results on several standard image classification datasets with a large number of classes.

External IDs:dblp:conf/copa/PensoGF25