Learning to Segment Multiple Organs from Multimodal Partially Labeled Datasets

Hong Liu, Dong Wei, Donghuan Lu, Jinghan Sun, Hao Zheng, Yefeng Zheng, Liansheng Wang

Abstract: Learning to segment multiple organs from partially labeled datasets can significantly reduce the burden of manual annotations. However, due to the large domain gap, learning from partially labeled datasets of different modalities has not been well addressed. In addition, the anatomic prior knowledge of various organs is spread in multiple datasets and needs to be more effectively utilized. This work proposes a novel framework for learning to segment multiple organs from multimodal partially labeled datasets (i.e., CT and MRI). Specifically, our framework constructs a cross-modal a priori atlas from training data, which implicitly contains prior knowledge of organ locations, shapes, and sizes. Based on the atlas, three novel modules are proposed to address the joint challenges of unlabeled organs and inter-modal domain gaps: 1) to better utilize unlabeled organs for training, we propose an atlas-guided pseudo-label refiner network to improve the quality of pseudo-labels; 2) we propose an atlas-conditioned modality alignment network for cross-modal alignment in the label space via adversarial training, forcing cross-modal segmentations of organs labeled in a different modality to match the atlas; and 3) to further align organ-specific semantics in the latent space, we introduce modal-invariant class prototype anchoring modules supervised by the refined pseudo-labels, encouraging domain-invariant features for each organ. Extensive experiments demonstrate the superior performance of our framework to existing state-of-the-art methods and the efficacy of its components.