Cross-Modality Segmentation by Self-supervised Semantic Alignment in Disentangled Content Space

Junlin Yang, Xiaoxiao Li, Daniel H. Pak, Nicha C. Dvornek, Julius Chapiro, Ming De Lin, James S. Duncan

Abstract: Deep convolutional networks have demonstrated state-of-the-art performance in a variety of medical image tasks, including segmentation. Taking advantage of images from different modalities has great clinical benefits. However, the generalization ability of deep networks on different modalities is challenging due to domain shift. In this work, we investigate the challenging unsupervised domain adaptation problem of cross-modality medical image segmentation. Cross-modality domain shift can be viewed as having two orthogonal components: appearance (modality) shift and content (anatomy) shift. Previous works using the popular adversarial training strategy emphasize the significant appearance/modality alignment caused by different physical principles while ignoring the content/anatomy alignment, which can be harmful for the downstream segmentation task. Here, we design a cross-modality segmentation pipeline, where self-supervision is introduced to achieve further semantic alignment specifically on the disentangled content space. In the self-supervision branch, in addition to rotation prediction, we also propose elastic transformation prediction as a new pretext task. We validate our model on cross-modality liver segmentation from CT to MR. Both quantitative and qualitative experimental results demonstrate that further semantic alignment through self-supervision can improve segmentation performance significantly, making the learned model more robust.