Valid Selection among Conformal Sets

Mahmoud Hegazy, Liviu Aolaritei, Michael I. Jordan, Aymeric Dieuleveut

Abstract:

Conformal prediction offers a distribution-free framework for constructing prediction sets
with coverage guarantees. In practice, multiple valid conformal prediction sets may be available, arising from different models or methodologies. However, selecting the most desirable
set, such as the smallest, can invalidate the coverage guarantees. To address this challenge,
we propose a stability-based approach that ensures coverage for the selected prediction
set. We extend our results to the online conformal setting, propose several refinements in
settings where additional structure is available, and demonstrate its effectiveness through
experiments.