Zero-Shot Medical Image Segmentation Based on Sparse Prompt Using Finetuned SAM

Tal Shaharabany, Lior Wolf

Keywords: zero-shot segmentation, multi-modal, detection

Abstract: Segmentation of medical images plays a critical role in various clinical applications, facilitat- ing precise diagnosis, treatment planning, and disease monitoring. However, the scarcity of annotated data poses a significant challenge for training deep learning models in the medical imaging domain. In this paper, we propose a novel approach for minimally-guided zero-shot segmentation of medical images using the Segment Anything Model (SAM), orig- inally trained on natural images. The method leverages SAM’s ability to segment arbitrary objects in natural scenes and adapts it to the medical domain without the need for labeled medical data, except for a few foreground and background points on the test image it- self. To this end, we introduce a two-stage process, involving the extraction of an initial mask from self-similarity maps and test-time fine-tuning of SAM. We run experiments on diverse medical imaging datasets, including AMOS22, MoNuSeg and the Gland segmen- tation (GlaS) challenge, and demonstrate the effectiveness of our approach.

Submission Number: 206