CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimation

Alyssa Unell, Noel C Codella, J. Samuel Preston, Peniel Argaw, Wen-wai Yim, Zelalem Gero, Cliff Wong, Eric Horvitz, Amanda Hall, Rachel Ruican Zhong, Jiachen Li, Shrey Jain, Mu Wei, Matthew P. Lungren, Hoifung Poon

Keywords: Guideline Compliance, Cancer, LLM Evaluation, Uncertainty, Decision Making, Unsupervised

TL;DR: LLM system for automated NCCN-concordant cancer treatment recommendations validated on 121 expert-annotated NSCLC cases, with proxy benchmarks achieving r=0.86 correlation and 0.847 AUROC accuracy verification.

Abstract: The National Comprehensive Cancer Network (NCCN) provides evidence-based guidelines for cancer treatment. Currently, translating complex patient presentations into guideline-compliant treatment recommendations is a time-intensive manual process requiring specialized expertise and is prone to error. Advances in large language model (LLM) capabilities promise to reduce the time required to generate treatment recommendations and improve accuracy. Toward this goal, we present an LLM agent-based approach to automatically generate guideline-concordant treatment trajectories for patients with non-small cell lung cancer (NSCLC). Our contributions are threefold. First, we construct a novel longitudinal dataset of 121 cases of NSCLC patients that includes clinical encounters, imaging findings, diagnostic results, and medical histories, each expertly annotated with the corresponding NCCN guideline trajectories by board-certified oncologists. Second, we demonstrate that existing LLMs possess domain-specific parametric knowledge that enables high-quality proxy benchmark generation for both model development and evaluation, achieving strong correlation (Spearman coefficient $r=0.88$, RMSE = $0.08$) with expert-annotated benchmarks. Third, we develop a hybrid approach combining expensive human annotations with model consistency information to create both the agent framework that predicts the relevant guidelines for a patient, as well as a meta-classifier that verifies prediction accuracy with calibrated confidence scores for treatment recommendations (AUROC=0.806), a critical feature to support regulatory compliance and error tradeoff customization. In summary, this work establishes a framework for developing clinically viable LLM-based guideline adherence systems that balance accuracy, interpretability, and regulatory requirements while reducing expert annotation costs. Our approach provides a scalable pathway toward automated clinical decision support that maintains the rigor necessary for healthcare deployment.

Submission Number: 54