Comparing SAM 2 and SAM 3 for Zero-Shot Segmentation of 3D Medical Data

Satrajit Chakrabarty, Ravi Soni

Keywords: Foundation models, Segment Anything Model, Zero-shot segmentation, SAM 2, SAM 3

TL;DR: We compare SAM 2 and SAM 3 under identical visual prompts on 16 public 3D medical datasets covering 54 structures and find SAM 3 is the stronger default for zero-shot segmentation, with SAM 2 only competitive for a few compact organs.

Abstract:

Foundation models, such as the Segment Anything Model (SAM), have heightened interest in promptable zero-shot segmentation. Although these models perform strongly on natural images, their behavior on medical data remains insufficiently characterized. While SAM 2 has been widely adopted for annotation in 3D medical workflows, the recently released SAM 3 introduces a new architecture that may change how spatial prompts are interpreted and propagated. Therefore, to assess whether SAM 3 can serve as an out-of-the-box replacement for SAM 2 for zero-shot segmentation of 3D medical data, we present the first controlled comparison of both models under purely spatial prompting, with concept mechanisms of SAM 3 disabled. We benchmark using a variety of prompting strategies on 16 public datasets (CT, MRI, Ultrasound, endoscopy) covering 54 anatomical structures, pathologies, and surgical instruments. We further quantify three failure modes: prompt-frame over-segmentation, over-propagation after object disappearance, and temporal retention of
well-initialized predictions. Our results show that SAM 3 provides stronger initialization than SAM 2 for click prompts and maintains higher Dice and more stable retention for complex, vascular, and soft-tissue anatomies. Under bounding box and mask, SAM 2 remains
competitive and often more conservative for compact organs by terminating tracks earlier and hallucinating less. The overall results position SAM 3 as the superior default choice for most medical segmentation tasks, while clarifying when SAM 2 remains a preferable
propagator.

Primary Subject Area: Segmentation

Secondary Subject Area: Foundation Models

Registration Requirement: Yes

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 26