Beyond Self-consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Staging

Chia-Hsuan Chang, Mary M. Lucas, Yeawon Lee, Christopher C. Yang, Grace Lu-Yao

Abstract: Pathologic cancer stage, crucial for treatment decisions, is often buried in unstructured pathology reports. This study investigates using pre-trained clinical LLMs for stage extraction, leveraging prompting techniques like chain-of-thought to enhance model transparency. While self-consistency methods further improve LLM performance, they can introduce inconsistencies in reasoning paths and predictions. We propose an ensemble reasoning approach, aiming for reliable cancer stage extraction. Utilizing an open-source clinical LLM on real-world reports, we demonstrate that the ensemble approach improves consistency and boosts performance, paving the way for utilizing LLMs in healthcare settings where reliability and interpretability are paramount.