Translating Classifier Scores into Clinical Impact: Calibrated Risk and Queueing Simulation for AI-Assisted Radiology Worklist Triage

Tirthajit Baruah, Punit Rathore

Keywords: AI triage, calibrated risk estimation, radiology workflow, queueing simulation, operational impact, deployable AI

TLDR: We link AI classifier performance to clinical workflow impact using calibrated risk and queueing simulation. On RSNA-ICH, score-based prioritization accelerates critical reads with minimal delay, enabling data-driven evaluation before deployment.

Abstract: Radiology worklists are typically processed first-in–first-out (FIFO) even when studies differ greatly in clinical urgency. We propose a pragmatic alternative: using calibrated probabilities of intracranial hemorrhage (ICH) to prioritize head CT exams for earlier reading. Using the public RSNA-ICH dataset, we train slice-level detectors, aggregate to exam-level, apply post-hoc calibration, and feed these scores into a transparent discrete-event simulator of the reading queue. The simulator quantifies how triage benefit reduction in median time-to-read (TTR) for ICH; scales with classifier AUC, workload (arrival rate), staffing, prevalence, and calibration. Across realistic loads, score-based prioritization yields substantial TTR reductions for ICH with minimal delay to non-ICH studies. We release a configuration-driven, reproducible pipeline that translates AI risk scores into operational metrics (minutes saved), enabling safe and data-driven evaluation before PACS/RIS deployment.

Submission Number: 38