Med-FastSAM: Improving Transfer Efficiency of SAM to Domain-Generalised Medical Image Segmentation

Yuxiang Luo, Qing Xu, Jinglei Feng, Guangwu Qian, Wenting Duan

Keywords: Transfer learning, Small foundation model, Medical image segmentation

Abstract: Medical image segmentation is a crucial computer vision task in medical image analysis. Recently, the Segment Anything Model (SAM) has made significant advancements in natural image segmentation. Despite current studies indicating the potential of SAM to revolutionise medical image segmentation using parameter-efficient fine-tuning techniques, it still faces three primary challenges. Firstly, these methods still rely on the large vision transformer of SAM, which is computationally expensive. Secondly, the point and box prompt modes of SAM demand manual annotations, which are time-consuming and expensive in medical scenarios and reduce their clinical applicability. Thirdly, SAM leverages large-size patches to predict masks, resulting in the loss of fine-grained details. To address these limitations, in this paper, we propose a fast-transferring architecture for adapting SAM to domain-generalised medical image segmentation, named Med-FastSAM. Specifically, we introduce a lightweight knowledge aggregation encoder that combines the distilled natural image knowledge with learned medical-specific information for producing feature representation. Moreover, we devise a coarse prompt module to automatically generate coarse masks for guiding segmentation decoding. Furthermore, we design a multi-scale feature decoder to produce precise segmentation masks. Eventually, extensive experiments on four benchmark datasets have been conducted to evaluate the proposed model. The result demonstrates that Med-FastSAM outperforms state-of-the-art methods without any manual prompts. Especially, our model shows excellent zero-shot domain generalisation performance by using only 15.45\% parameters compared to the standard SAM. The code for our work and more technical details can be found at https://github.com/GalacticHogrider/Med-FastSAM.

Submission Number: 36