SynSUM -- Synthetic Benchmark with Structured and Unstructured Medical Records

Paloma Rabaey, Henri Arno, Stefan Heytens, Thomas Demeester

Keywords: electronic health records, synthetic benchmark, structured and unstructured data, bayesian network, large language model

TL;DR: SynSUM is a synthetic benchmark of 10,000 artificial patient records linking structured tabular features (symptoms, diagnoses, and background, generated by an expert-defined Bayesian network) to unstructured clinical notes (generated by GPT-4o).

Abstract: We present the SynSUM benchmark, a synthetic dataset linking unstructured clinical notes to structured background variables. The dataset consists of 10,000 artificial patient records containing tabular variables (like symptoms, diagnoses and underlying conditions) and related notes describing the fictional patient encounter in the domain of respiratory diseases. The tabular portion of the data is generated through a Bayesian network, where both the causal structure between the variables and the conditional probabilities are proposed by an expert based on domain knowledge. We then prompt a large language model (GPT-4o) to generate a clinical note related to this patient encounter, describing the patient symptoms and additional context. We conduct both an expert evaluation study to assess the quality of the generated notes, as well as running some simple predictor models on both the tabular and text portions of the dataset, forming a baseline for further research. The SynSUM dataset is primarily designed to facilitate research on clinical information extraction in the presence of tabular background variables, which can be linked through domain knowledge to concepts of interest to be extracted from the text -- the symptoms, in the case of SynSUM. Secondary uses include research on the automation of clinical reasoning over both tabular data and text, causal effect estimation in the presence of tabular and/or textual confounders, and multi-modal synthetic data generation. The dataset will be accessible on Github upon publication.

Submission Number: 9