Valid Selection among Conformal Sets

Mahmoud Hegazy, Liviu Aolaritei, Michael I. Jordan, Aymeric Dieuleveut

Keywords: Conformal Prediction, Selection after Inference, Uncertainty Quantification

TL;DR: When multiple conformal predictors are available, we allow data-dependent selection without the loss of coverage.

Abstract: Conformal prediction offers a distribution-free framework for constructing prediction sets with coverage guarantees. In practice, multiple valid conformal prediction sets may be available, arising from different models or methodologies. However, selecting the most desirable set, such as the smallest, can invalidate the coverage guarantees. To address this challenge, we propose a stability-based approach that ensures coverage for the selected prediction set. We extend our results to the online conformal setting, propose several refinements in settings where additional structure is available, and demonstrate its effectiveness through experiments.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 17817