PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates

Junjie Shi, Caozhi Shang, Zhaobin Sun, Li Yu, Xin Yang, Zengqiang Yan

Abstract: Incomplete multi-modal image segmentation is a fundamental task in medical imaging to refine deployment efficiency when only partial modalities are available. However, the common practice that complete-modality data is visible during model training is far from realistic, as modalities can have imbalanced missing rates in clinical scenarios. In this paper, we, for the first time, formulate such a challenging setting and propose Preference-Aware Self-diStillatION (PASSION) for incomplete multi-modal medical image segmentation under imbalanced missing rates. Specifically, we first construct pixel-wise and semantic-wise self-distillation to balance the optimization objective of each modality. Then, we define relative preference to evaluate the dominance of each modality during training, based on which to design task-wise and gradient-wise regularization to balance the convergence rates of different modalities. Experimental results on two publicly available multi-modal datasets demonstrate the superiority of PASSION against existing approaches for modality balancing. More importantly, PASSION is validated to work as a plug-and-play module for consistent performance improvement across different backbones. Code will be available upon acceptance.

Primary Subject Area: [Content] Vision and Language

Secondary Subject Area: [Content] Multimodal Fusion

Relevance To Conference: In this paper, we, for the first time, identify and formulate a more practical yet under-explored multi-modal segmentation task, namely incomplete multi-modal medical image segmentation with imbalanced missing rates. Different from existing research assuming balanced modality distributions during training, modalities can have varying missing rates, which is more realistic but challenging. To address this, we propose a plug-and-play module named PASSION. PASSION adopts pixel-wise and semantic-wise self-distillation to balance modality-specific optimization objectives and penalizes task-wise and gradient-wise regularization to balance the convergence rates of different modalities. Experimental results on two publicly available multi-modal datasets demonstrate that PASSION outperforms existing approaches for modality balancing given imbalanced missing rates and is highly extendable to various backbones for consistent performance improvement.

Submission Number: 4561