On Training-Conditional Conformal Prediction and Binomial Proportion Confidence Intervals

Rudi Coppola, Manuel Mazo Espinosa

Abstract:

Estimating the expectation of a Bernoulli random variable based on $N$ independent trials is a classical problem in statistics, typically addressed using Binomial Proportion Confidence Intervals (BPCI). In the control systems community, many critical tasks—such as certifying the statistical safety of dynamical systems—can be formulated as BPCI problems.

Conformal Prediction (CP), a distribution-free technique for uncertainty quantification, has gained significant attention in recent years and has been applied to various control systems problems, particularly to address uncertainties in learned dynamics or controllers. A variant known as training-conditional CP was recently employed to tackle the problem of safety certification.

In this note, we highlight that the use of training-conditional CP in this context does not provide valid safety guarantees. We demonstrate why CP is unsuitable for BPCI problems and argue that traditional BPCI methods are better suited for statistical safety certification.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: We polished Section 4 to improve the readability as suggested by the AE. In particular, we simplified some sentences, divided some paragraphs, and used enumerations to highlight the takeaways in Section 4, while preserving the additional clarifying content suggested by the reviewers. We hope that the AE and the reviewers are satisfied with the final version.

Code: https://github.com/Ru-Co-La/training-conditional-conformal-prediction.git

Assigned Action Editor: ~Zheng_Wen1

Submission Number: 3760