Class-Conditional Conformal Prediction with Many Classes

Tiffany Ding, Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, Ryan Tibshirani

Keywords: conformal prediction, uncertainty quantification, class imbalance

TL;DR: We tackle the problem of producing prediction sets with good class-conditional coverage in the challenging limited-data, many-classes classification setting.

Abstract:

Standard conformal prediction methods provide a marginal coverage guarantee,
which means that for a random test point, the conformal prediction set contains
the true label with a user-specified probability. In many classification
problems, we would like to obtain a stronger guarantee--that for test points
of a specific class, the prediction set contains the true label with the
same user-chosen probability. For the latter goal, existing conformal prediction
methods do not work well when there is a limited amount of labeled data per
class, as is often the case in real applications where the number of classes is
large. We propose a method called clustered conformal prediction that
clusters together classes having "similar" conformal scores and performs
conformal prediction at the cluster level. Based on empirical evaluation across
four image data sets with many (up to 1000) classes, we find that clustered
conformal typically outperforms existing methods in terms of class-conditional
coverage and set size metrics.

Submission Number: 10314