Sample-Conditional Coverage in Split-Conformal Prediction

John Duchi

Keywords: Conformal prediction, predictive inference, coverage

TL;DR: Develops split conformal methods that achieve approximately conditional coverage on new predictions

Abstract: We revisit the problem of constructing predictive confidence sets for which we wish to obtain some type of conditional validity. We provide new arguments showing how ``split conformal'' methods achieve near desired coverage levels with high probability, a guarantee conditional on the validation data rather than marginal over it. In addition, we directly consider (approximate) conditional coverage, where, e.g., conditional on a covariate $X$ belonging to some group of interest, we seek a guarantee that a predictive set covers the true outcome $Y$. We show that the natural method of performing quantile regression on a held-out (validation) dataset yields minimax optimal guarantees of coverage in these cases. Complementing these positive results, we also provide experimental evidence highlighting work that remains to develop computationally efficient valid predictive inference methods.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 26688