LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models

Jingteng Li, Alexander Capstick, Louise Rigny, Iona Biggart, Neil J Sebire, Payam Barnaghi

Keywords: Electronic healthcare records, Time series, Large lanugage model, Contrastive learning

TL;DR: LLM-powered Multimodal clinical foundation for learning entangled representations of EHR event sequences and clinical time series via contrastive learning.

Abstract:

Recent research in clinical machine learning, focusing on the intensive care unit (ICU), has shifted from bespoke supervised models to foundation models, utilising Large Language Models (LLMs). Here, LLMs are fine-tuned on mixtures of complex clinical data modalities, useful for various downstream tasks. However, existing methods do not sufficiently explore the shared temporal structure between the events on Electronic Health Records (EHRs) and clinical Time Series (TS) observations. This limitation potentially leads to less robust and adaptive clinical foundation models, resulting in reduced performance on downstream tasks. To fully exploit this temporal structure, we propose LLM4EHR, a new clinical foundation model trained on ICU data.
Combining pre-trained LLMs with additional trainable layers, we fine-tune our model to temporally align the EHR and TS modalities. For this, we propose a regularised contrastive objective to jointly learn representations of EHRs and clinical TS.
Supported by an ablation study, we find that embeddings from LLM4EHR improve performance on various downstream clinical tasks with competitive performance in a few-shot setting. Further, we empirically demonstrate that LLM4EHR learns transferable clinical TS embeddings that can be deployed to new cohorts with minimal performance loss. These findings provide a step towards building more generalisable and performant clinical foundation models.

Primary Area: learning on time series and dynamical systems

Submission Number: 12326