When SAM2 Learns to Prompt Itself: Tuning-Free Inference and Self-Distillation for Medical Image Segmentation

Liyuqiu Huang, Gaole Dai, Shanghang Zhang

Keywords: Medical Image Segmentation, Self-distillation, Prompt-based segmentation

Abstract:

The Segment Anything Model (SAM) family has established a paradigm of prompt-based interactive segmentation, attracting substantial attention. Prior research has indicated the possibility of enhancing the performance of SAM2 by modifying inference strategies. For instance, SAMURAI utilizes Kalman-based test-time optimization, and FS-MedSAM2 applies few-shot reference learning.
In this paper, we delve into this emerging trend within the realm of high-dimensional medical image segmentation tasks. Firstly, we put forward three novel inference modules:
(1) Breadth-First Search memory collection, which gives precedence to central slices containing high-information content;
(2) Multi-dimensional result voting, designed to stabilize predictions and mitigate bias;
(3) a self-growing prompt mechanism that generates new visual prompts during the segmentation process.
Our experimental outcomes surpassed all baseline tuning-free methods, and the relative gain is on par with fine-tuning. Additionally, we explore the self-correction potential of SAM through tuning. We also introduced a learnable meta-prompter to extend the capabilities of our inference modules. We carried out multi-round self-distillation based on the outputs of the proposed inference framework. In most scenarios, distilled model's performance can be enhanced without additional annotation.
Collectively, these results highlight a practical approach for implementing foundation models in real-world medical segmentation tasks with minimal annotation requirements.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: true

Submission Guidelines: true

Anonymous Url: true

No Acknowledgement Section: true

Submission Number: 7047