SIGxCL: A Signal-Image-Graph Cross-Modal Contrastive Learning Framework for CVD Diagnosis Based on Internet of Medical Things

Huaicheng Zhang, Wenhan Liu, Zhoutong Li, Jiguang Shi, Sheng Chang, Hao Wang, Jin He, Qijun Huang

Abstract: Recently, contrastive learning (CL) has garnered wide interest because it enables unsupervised pretraining to alleviate conventional deep learning methods’ strong reliance on artificial labels. While CL-based methods have been applied to cardiovascular disease (CVD) diagnosis with noninvasive electrocardiogram (ECG), most of these methods are limited within the 1-D signal modality and primarily focus on temporal features like amplitude and time sequence. The morphological features derived from clinically significant image-like ECGs are ignored. Furthermore, the relationships among different leads are neglected as well, describing the activities and interaction of various heart regions that are essential in CVD diagnosis and lesion localization. To address these limitations, this work proposes a novel cross-modal CL framework named signal–image–graph cross-modal contrastive learning (SIGxCL), which represents and jointly analyzes ECGs in signal, image, and graph modalities. Crucial for CL, modality-specific transformations are introduced for ECGs in the three modalities. SIGxCL enables signal–image–graph correspondence by maximizing the agreement of the accordant cross-modal ECGs in the invariant space. Consequently, SIGxCL could capture and leverage temporal, morphological, and spatially physiological features simultaneously. Compared to random initial and conventional supervised methods, SIGxCL achieves remarkable enhancements. Considering the best performances of the existing CL-based methods, SIGxCL outperforms them by up to 4.72%, 9.41%, and 4.31% across three data sets. SIGxCL is designed to be compatible with Internet of Medical Things (IoMT) and can be deployed on resource-limited portable devices. The deployment includes pretraining, online/offline tuning, and real-time inference modules. In conclusion, SIGxCL demonstrates superior performance and provides a promising approach for real-time IoMT-based diagnosis.