On the Importance of Clinical Notes in Multi-modal Learning for EHR Data

Severin Husmann, Hugo Yèche, Gunnar Ratsch, Rita Kuznetsova

Keywords: Clinical notes, ICU data, EHR, Multi-modal learning, Interpretability

TL;DR: We analyze performance improvements obtained from jointly using clinical notes and EHR data over uni-modal approaches in ICU patient monitoring.

Abstract: Understanding deep learning model behavior is critical to accepting machine learning-based decision support systems in the medical community. Previous works have shown that jointly using clinical notes with electronic health record (EHR) data improved predictive performance for patient monitoring in the intensive care unit (ICU). In this work, we explore the underlying reasons for these improvements. While relying on a basic attention-based model to allow for interpretability, we first confirm that performance significantly improves over state-of-the-art EHR data models when combining EHR data and clinical notes. We then provide an analysis showing improvements arise almost exclusively from a subset of notes containing broader context on patient state rather than clinician notes. We believe such findings highlight deep learning models for EHR data to be more limited by partially-descriptive data than by modeling choice, motivating a more data-centric approach in the field.