Learning Visual Features by Colorization for Slide-Consistent Survival Prediction from Whole Slide Images

Lei Fan, Arcot Sowmya, Erik Meijering, Yang Song

Abstract: Recent deep learning techniques have shown promising performance on survival prediction from Whole Slide Images (WSIs). These methods are often based on multiple-step frameworks including patch sampling, feature extraction, and feature aggregation. However, feature extraction typically relies on handcrafted features or Convolutional Neural Networks (CNNs) pretrained on ImageNet without fine-tuning, thus leading to suboptimal performance. Besides, to aggregate features, previous studies focus on WSI-level survival prediction but ignore the heterogeneous information that is present in multiple WSIs acquired for the same patient. To address the above challenges, we propose a survival prediction model that exploits heterogeneous features at the patient-level. Specifically, we introduce colorization as the pretext task to train the CNNs which are tailored for extracting features from patches of WSIs. In addition, we develop a patient-level framework integrating multiple WSIs for survival prediction with consistency and ranking losses. Extensive experiments show that our model achieves state-of-the-art performance on two large-scale public datasets.