Self-Prompt SAM: Automatic Prompt SAM Adaptation for Medical Image Segmentation
Keywords: SAM, Adapter, Medical image segmentation
Abstract: The Segment Anything Model (SAM), a prompt-driven foundation model for natural image segmentation, has demonstrated impressive zero-shot performance and brought a range of unexplored capabilities to natural image segmentation tasks. However, as a very important branch of image segmentation, the performance of SAM remains uncertain when it is applied to medical image segmentation due to the significant differences between natural images and medical images. Meanwhile, it is harsh to meet the requirements of extra prompts provided, such as points or boxes to specify medical regions, since medical knowledge is not expected from users.
In this paper, we aim to adapt pre-trained SAM models worked on from 2D natural images to 3D medical images without any prompts provided. Through the analysis of SAM models, we propose a novel self-prompt SAM adaptation framework for medical image segmentation, named Self-Prompt SAM. We designed a multi-scale prompt generator combined with the image encoder in SAM to generate auxiliary masks. Then, we use the auxiliary masks to generate bounding boxes as box prompts and utilize Distance Transform to select the points farthest from the boundary as point prompts. Meanwhile, we designed a 3D depth-fused adapter (DfusedAdapter) and injected the DFusedAdapter into each transformer block in the image encoder and mask decoder to enable pre-trained 2D SAM models to extract 3D information and adapt to 3D medical images.
Extensive experiments demonstrate that our method outperforms existing state-of-the-art approaches on two challenging public ACDC and Synapse datasets.
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4254
Loading