Deep Neural Network for Lung Adenocarcinoma Subtype from Multimodal Fusion of Imaging and Clinical Data

Chenchen Fan, Liya Liu, Yuxuan Wang, Danna Li, Qinghua Liang, Ahmed Elazab, Zhou Liu, Jiahang Hu, Yuan Tian, Yongquan Zhang, Changmiao Wang

Abstract: Accurate identification of lung adenocarcinoma subtype is crucial for selecting appropriate treatment plans. However, challenges such as integrating diverse data, subtype similarities, and capturing contextual features hinder precise differentiation. To address these challenges, we propose a deep neural network model that integrates CT images, annotated lesion bounding boxes, and electronic health records. Initially, the model leverages combining bounding boxes and CT scans to produce enhanced CT images with detailed lesion location information. These enhanced CT images then undergo feature extraction through a vision transformer module. In addition to imaging data, the model incorporates clinical information, encoded using a fully connected encoder. Features extracted from both CT and clinical data are optimized for cosine similarity using a CLIP module, ensuring their cohesive integration. To further blend features from different modalities, we devise an attention-based feature fusion module, known as DFF, to harmonize them into a unified representation. This integrated feature set is then input to a classifier that effectively differentiates among the three types of adenocarcinomas. To alleviate the effect of unbalanced classes, we incorporate contrastive learning loss and focal loss, which enhance feature representation and improve model performance. Our model achieves a superior accuracy of 81.42% and an area under the curve of 0.9120 on the validation set, significantly outperforming other recent multimodal classification methods. The code is available at https://github.com/fancccc/LungCancerDC.