Temporally-Extended Prompts Optimization for SAM in Interactive Medical Image Segmentation

Chuyun Shen, Wenhao Li, Ya Zhang, Xiangfeng Wang

Keywords: Interactive Medical Image Segmentation, Reinforcement Learning, Segment Anything Model

Abstract: The $\textit{Segmentation Anything Model}$ (SAM) has recently emerged as a foundation model for addressing image segmentation. Owing to the intrinsic complexity of medical images and the high annotation cost, the medical image segmentation (MIS) community has been encouraged to investigate SAM's zero-shot capabilities to facilitate automatic annotation. Inspired by the extraordinary accomplishments of $\textit{interactive}$ medical image segmentation (IMIS) paradigm, this paper focuses on assessing the potential of SAM's zero-shot capabilities within the IMIS paradigm to amplify its benefits in the MIS domain. Regrettably, we observe that SAM's vulnerability to prompt modalities (e.g., points, bounding boxes) becomes notably pronounced in IMIS. This leads us to develop a mechanism that adaptively offers suitable prompt modalities for human experts. We refer to the mechanism above as $\textit{temporally-extended prompts optimization}$ (TEPO) and model it as a Markov decision process, solvable through reinforcement learning. Numerical experiments on the standardized benchmark $\texttt{Brats2020}$ demonstrate that the learned TEPO agent can further enhance SAM's zero-shot capability in the MIS context.

Submission Number: 33