False Coverage Proportion Control for Conformal Prediction

Alexandre Blain, Bertrand Thirion, Pierre Neuvial

TL;DR: We develop an aggregation-based method to achieve tight control of the False Coverage Proportion in a transductive conformal inference framework.

Abstract:

Split Conformal Prediction (SCP) provides a computationally efficient way to construct confidence intervals in prediction problems. Notably, most of the theory built around SCP is focused on the single test point setting. In real-life, inference sets consist of multiple
points, which raises the question of coverage guarantees for many points simultaneously. While *on average*, the False Coverage Proportion (FCP) remains controlled, it can fluctuate strongly around its mean, the False Coverage Rate (FCR). We observe that when a
dataset is split multiple times, classical SCP may not control the FCP in a majority of the splits. We propose CoJER, a novel method that achieves sharp FCP control in probability for conformal prediction, based on a recent characterization of the distribution of conformal $p$-values in a transductive setting. This procedure incorporates an aggregation scheme which provides robustness with respect to modeling choices. We show through extensive real data experiments that CoJER provides FCP control while standard SCP does not. Furthermore, CoJER yields shorter intervals than the *state-of-the-art method* for FCP control and only slightly larger intervals than standard SCP.

Lay Summary: Conformal prediction is a popular method for producing confidence intervals that work with any predictive model, offering a reliable measure of uncertainty. While the standard approach, Split Conformal Prediction (SCP), provides guarantees for individual predictions, it often fails to control errors when applied to many predictions at once. We propose CoJER, a new method that ensures accurate error control across multiple predictions by combining recent theoretical insights with a robust aggregation scheme. CoJER consistently achieves tighter and more reliable confidence intervals than existing methods, making it a practical tool for large-scale applications in areas like healthcare, finance, and scientific analysis.

Link To Code: https://github.com/sanssouci-org/CoJER-paper

Primary Area: General Machine Learning

Keywords: Conformal prediction, multiple testing, False Discovery control

Submission Number: 6862