Conformal Tradeoffs: Operational Profiles Beyond Coverage

Abstract:

Conformal prediction gives exact finite-sample coverage guarantees under
exchangeability, but deployed systems are judged by more than coverage
alone. For a fixed calibrated rule reused over a finite operational
window, stakeholders also care about deployment-facing quantities such as
commitment frequency, deferral, and decisive error exposure. These are
not determined by coverage: calibration choices with similar coverage can
still induce materially different operational profiles.

We study this characterization gap in a scoped setting: binary
split conformal prediction under exchangeability with a fixed deployed
rule. We introduce the Small-Sample Beta Correction (SSBC) which gives finite-sample coverage
semantics for the deployed rule: it inverts the Beta/Beta--Binomial law
governing calibration-conditional coverage to map a user request
$(\alpha^\star,\delta)$ to the least conservative calibration grid point
with calibration-conditional PAC semantics for the realized deployed rule.
Calibrate-and-Audit then fixes the rule by
calibration and uses an independent audit split to estimate the induced
region--class label table, a reusable summary from which deployment-facing
Key Performance Indicators (KPIs) follow by projection. Under this design,
fixed operational rates admit exact finite-sample Binomial inference,
while Beta--Binomial envelopes serve as practical predictive summaries for
future windows. The induced partition also exposes regime boundaries,
Pareto-relevant tradeoffs, and inverse-pricing questions for fixed
downstream conventions.

Simulations validate the SSBC semantics and compare audit-based summaries
with leave-one-out planning proxies; molecular toxicity data provide an
audit-based empirical example, and a solubility case study illustrates scenario
planning once coverage semantics are fixed.

Submission Type: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: N/A

Assigned Action Editor: ~Olawale_Elijah_Salaudeen1

Submission Number: 7879