Trans-SAM: Transfer Segment Anything Model to medical image segmentation with Parameter-Efficient Fine-Tuning

Yanlin Wu, Zhihong Wang, Xiongfeng Yang, Hong Kang, Along He, Tao Li

Abstract: Recently, the Segment Anything Model (SAM) has gained substantial attention in image segmentation due to its remarkable performance. It has demonstrated impressive zero-shot capabilities through interactive segmentation. However, due to the predominance of natural images in the training data, it often exhibits unsatisfactory performance when directly applied to medical image segmentation tasks. Training SAM from scratch in the medical domain is computationally resource-intensive and data-demanding. Additionally, when fine-tuning pre-trained SAM, the limited quantity of medical data will potentially lead to catastrophic forgetting. To address these issues, we proposed Trans-SAM, which utilizes Parameter-Efficient Fine-Tuning (PEFT) to transfer SAM into the medical image segmentation. The key novelties include the Intuitive Perceptual Fine-tuning (IPF) adapter, which directly integrates input image features into each layer of the encoder, and the Multi-scale Domain Transfer (MDT) adapter, a convolution-based mechanism designed to infuse inductive biases into the SAM. With our method, the general features extracted by pre-trained SAM can be better transferred to the medical domain. Moreover, our method can achieve satisfactory performance with a minimal amount of data during the fine-tuning. We conducted extensive evaluations on six medical datasets including different organs and modalities. The experimental results demonstrate that Trans-SAM shows superior performance compared to state-of-the-art PEFT methods. It significantly improves the performance of SAM for medical image segmentation. The code will be available at https://github.com/wuyanlin-wyl/Trans-SAM.