CLIP-Driven Universal Model for Partially Labeled Organ and Pan-cancer Segmentation

Jie Liu, Alan Yuille, Yucheng Tang, Zongwei Zhou

Keywords: Partial Label; Organ Segmentation; Universal Model

Abstract: Automatic multi-organ segmentation in medical image anal- ysis is a crucial task with various applications in computer-aided diag- nosis and treatment. Convolutional neural networks (CNNs) have shown success in segmenting abdominal organs in CT images, but challenges arise due to complex morphology, low tissue contrast, and limited fully labeled datasets. Learning from partially labeled datasets has emerged as a promising solution. However, assembling partially annotated datasets presents formidable challenges, including background inconsistency and label orthogonality. To address these challenges, this study introduces the Universal Model, which incorporates text embedding and a masked back-propagation mechanism with binary segmentation masks. A revised label taxonomy is maintained, and binary segmentation masks are gen- erated for each class during image pre-processing. The CLIP-based label encoding enhances the anatomical structure of the universal model’s fea- ture embedding, and loss is only computed for classes with available labels.

Submission Number: 30