Multi-modality Fusion Based Lung Cancer Survival Analysis with Self-supervised Whole Slide Image Representation Learning

Yicheng Wang, Ye Luo, Bo Li, Xiaoang Shen

Abstract: Whole slide pathological images (WSIs) are the gold standard for lung cancer prognosis. However, due to their high resolution and limited annotations, lung cancer survival analysis based on WSIs becomes a challenging task. Some recent methods that fuse WSI and other modalities have achieved certain results. However, these methods tend to focus on integrating gene related information while overlooking relatively easily obtainable clinical variables and often rely on labor-intensive ROI annotations. In this work, we propose a novel framework for lung cancer survival analysis, which obviates the need for ROI annotations and fully exploits WSIs and clinical information by introducing a multi-modality fusion module and multi-task learning. We also utilizes self-supervised learning to eliminate the heterogeneity between WSIs and natural images. Experimental results via 5-fold cross-validation on 1,225 WSIs from 444 patients from NLST validate the state-of-the-art performance of our proposed method.