Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models

Intae Moon, Ying Jin, Marinka Zitnik

Keywords: structured electronic health records, foundation models, conformal prediction, selective prediction, uncertainty quantification, clinical decision support

TL;DR: StratCP is an error-controlled act-or-defer framework that identifies when EHR foundation model predictions are reliable enough to use and when to defer with calibrated follow-up uncertainty.

Abstract:

Clinical deployment of foundation models requires decision policies that operate under explicit error budgets, such as a cap on false-positive clinical calls. Strong average accuracy alone does not guarantee safety: errors can concentrate among patients selected for action, leading to harm and inefficient use of healthcare resources.

Here we develop StratCP, a stratified conformal framework that turns foundation model predictions into decision-ready outputs through error-controlled selection and calibrated deferral. StratCP first selects a subset of patients for immediate clinical action while controlling false discovery rates at a user-specified level. For the remaining patients, it returns prediction sets that achieve target coverage conditional on deferral, supporting confirmatory testing or expert review.

We evaluate StratCP on Electronic Health Record (EHR) foundation model predictions across EHRSHOT tasks spanning operational outcomes, assignment of new diagnoses, and anticipating lab test results, including length of stay, pancreatic cancer, and thrombocytopenia severity prediction. Across tasks, StratCP controls class-specific false discovery rates among selected predictions and provides valid, selection-conditional coverage for deferred patients, with the largest gains on rarer and higher-stakes classes.

StratCP establishes error-controlled decision policies for safe deployment of medical foundation models.

Submission Number: 100