Examination Feedback Simulation: Beyond Static and Unique Clinical Trajectories

Renwei Xiao, Wen He, Hongzhou Yu, Qingqiu Li, Yuejie Zhang, Rui Feng, Xiaobo Zhang

Keywords: LLM for Healthcare, Virtual Patient, LLM-based Simulation

TL;DR: This paper investigates LLMs' limitations and potential in simulating clinical examination feedback.

Abstract: Large language models (LLMs) have shown significant promise in healthcare applications. To better mirror real-world settings for LLM evaluation, dynamic longitudinal diagnosis-and-treatment simulation with virtual patients has recently emerged as a focal point of research. However, existing simulation frameworks are constrained by the limitation of clinical trajectory uniqueness, where virtual patients can only provide feedback based on information available in static electronic health records (EHRs). This limitation leads to simulation failures when unrecorded but medically sound examinations are ordered. In this paper, we formulate the task of Examination Feedback Simulation to address this limitation, which aims to dynamically augment the unique trajectory by simulating medically plausible examination results in response to clinical orders. To support this largely unexplored research, we construct ClinTrack, a dataset curated from MIMIC-IV. ClinTrack is organized in a hierarchical, chronologically-ordered structure to facilitate sequential clinical tasks. We further propose a structure-aware evaluation metric SimScore to quantitatively assess the quality of simulated results, which shows promising initial alignment with expert judgment. Building on this framework, we develop ClinSim, a new generative model specifically designed for this task. Experiments demonstrate that our 4-billion-parameter ClinSim model significantly outperforms flagship models up to 235B parameters on this task, achieving an improvement of over 10 percentage points in SimScore, providing a critical foundation for creating more dynamic and realistic virtual patients.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Submission Number: 12549