Adaptive Coverage Policies in Conformal Prediction

Etienne Gauthier, Francis Bach, Michael I. Jordan

TL;DR: We propose a neural-network-based adaptive conformal prediction method that selects data-dependent coverage levels for each example, producing more informative prediction sets while maintaining valid statistical guarantees.

Abstract: Traditional conformal prediction methods construct prediction sets such that the true label falls within the set with a user-specified coverage level. However, poorly chosen coverage levels can result in uninformative predictions, either producing overly conservative sets when the coverage level is too high, or empty sets when it is too low. Moreover, the fixed coverage level cannot adapt to the specific characteristics of each individual example, limiting the flexibility and efficiency of these methods. In this work, we leverage recent advances in e-values and post-hoc conformal inference, which allow the use of data-dependent coverage levels while maintaining valid statistical guarantees. We propose to optimize an adaptive coverage policy by training a neural network using a leave-one-out procedure on the calibration set, allowing the coverage level and the resulting prediction set size to vary with the difficulty of each individual example. We support our approach with theoretical coverage guarantees and demonstrate its practical benefits through a series of experiments.

Code Dataset Promise: Yes

Code Dataset Url: https://github.com/GauthierE/adaptive-coverage-policies

Format Confirmation: I agree that I have read and followed the formatting instructions for the camera ready version.

Submission Number: 1405