impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction

Maria Boyko, Beliaeva Aleksandra, Dmitrii Kornilov, Alexander V. Bernstein, Maksim G. Sharaev

Keywords: Multimodal Learning, Multimodal pre-training strategy, Cancer Survival Prediction, Imputation of missing modalities

Abstract: The use of diverse modalities, such as omics, medical images, and clinical data can not only improve the performance of prognostic models but also deepen an understanding of disease mechanisms and facilitate the development of novel treatment approaches. However, medical data are complex, often incomplete, and contains missing modalities, making effective handling its crucial for training multimodal models. We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy. It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches. Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for cancer survival prediction using TCGA dataset, integrating three modalities: genetic (DNAm, RNA-seq), imaging (WSI), and clinical data. By addressing missing data during pre-training and enabling efficient resource utilization, impuTMAE surpasses prior multimodal approaches, achieving state-of-the-art performance in cancer survival prediction.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 11476