Does Multimodality Help in Deep Learning-Based Structural Heart Disease Detection?

Young Sang Choi, Shalmali Joshi, Pierre Elias

Keywords: multimodal learning, cardiology, structural heart disease, deep learning

Abstract: Structural heart disease (SHD) is typically diagnosed using transthoracic echocardiograms (TTEs), a modality underutilized in the United States. We investigate what combination of common clinical modalities in electrocardiograms (ECGs), posteroanterior view chest X-rays, and structured electronic health record (EHR) data can detect SHD labels generated with an TTE unseen by the model. Our experiments show that ECG-based models in both unimodal and multimodal settings performed best and the inclusion of additional modalities can give a marginal performance improvement with a late-fusion approach.

Submission Number: 155