Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training

Jeya Maria Jose Valanarasu, Yucheng Tang, Dong Yang, Ziyue Xu, Can Zhao, Wenqi Li, Vishal M. Patel, Bennett Allan Landman, Daguang Xu, Yufan He, Vishwesh Nath

Keywords: Self-supervised Learning, Image Segmentation

Abstract: Harnessing the power of pre-training on large-scale datasets like ImageNet forms a funda- mental building block for the progress of representation learning-driven solutions in com- puter vision. Medical images are inherently different from natural images as they are acquired in the form of many modalities (CT, MR, PET, Ultrasound etc.) and contain granulated information like tissue, lesion, organs etc. These characteristics of medical im- ages require special attention towards learning features representative of local context. In this work, we focus on designing an effective pre-training framework for 3D radiology im- ages. First, we propose a new masking strategy called local masking where the masking is performed across channel embeddings instead of tokens to improve the learning of local feature representations. We combine this with classical low-level perturbations like adding noise and downsampling to further enable low-level representation learning. To this end, we introduce Disruptive Autoencoders, a pre-training framework that attempts to re- construct the original image from disruptions created by a combination of local masking and low-level perturbations. We curate a large-scale dataset to enable pre-training of 3D medical radiology images (MRI and CT). The proposed pre-training framework is tested across multiple downstream tasks and achieves state-of-the-art performance. Notably, our proposed method tops the public test leaderboard of BTCV multi-organ segmentation chal- lenge. Our code can be found here: https://github.com/Project-MONAI/research-contributions/tree/main/DAE.

Submission Number: 58