Memory Replay for Continual Medical Image Segmentation Through Atypical Sample Selection

Sutanu Bera, Vinay Ummadi, Debashis Sen, Subhamoy Mandal, Prabir Kumar Biswas

Abstract: Medical image segmentation is critical for accurate diagnosis, treatment planning and disease monitoring. Existing deep learning-based segmentation models can suffer from catastrophic forgetting, especially when faced with varying patient populations and imaging protocols. Continual learning (CL) addresses this challenge by enabling the model to learn continuously from a stream of incoming data without the need to retrain from scratch. In this work, we propose a continual learning-based approach for medical image segmentation using a novel memory replay-based learning scheme. The approach uses a simple and effective algorithm for image selection to create the memory bank by ranking and selecting images based on their contribution to the learning process. We evaluate our proposed algorithm on three different problems and compare it with several baselines, showing significant improvements in performance. Our study highlights the potential of continual learning-based algorithms for medical image segmentation and underscores the importance of efficient sample selection in creating memory banks.