Missing as Masking: Arbitrary Cross-Modal Feature Reconstruction for Incomplete Multimodal Brain Tumor Segmentation

Zhilin Zeng, Zelin Peng, Xiaokang Yang, Wei Shen

Abstract: Automatic brain tumor segmentation using multimodal MRI images is a critical task in medical imaging. A complete set of multimodal MRI images for a subject offers comprehensive views of brain tumors, and thus providing ideal tumor segmentation performance. However, acquiring such modality-complete data for every subject is frequently impractical in clinical practice, which requires a segmentation model to be able to 1) flexibly leverage both modality-complete and modality-incomplete data for model training, and 2) prevent significant performance degradation in inference if certain modalities are missing. To meet these two demands, in this paper, we propose M\(^3\)FeCon (Missing as Masking: arbitrary cross-Modal Feature ReConstruction) for incomplete multimodal brain tumor segmentation, which can learn approximate modality-complete feature representations from modality-incomplete data. Specifically, we treat missing modalities also as masked modalities, and employ a strategy similar to Masked Autoencoder (MAE) to learn feature-to-feature reconstruction across arbitrary modality combinations. The reconstructed features for missing modalities act as supplements to form approximate modality-complete feature representations. Extensive evaluations on the BraTS18 dataset demonstrate that our method achieves state-of-the-art performance in brain tumor segmentation with incomplete modalities, especicall in enhancing tumor with 4.61% improvement in terms of Dice score.