SAM 3: Segment Anything with Concepts

Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris Coll-Vinent, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras et al. (18 additional authors not shown)

Keywords: foundation models, open vocabulary segmentation, semantic instance segmentation, object tracking

TL;DR: We present a strong model and challenging benchmark to advance open-vocabulary concept segmentation in images and videos.

Abstract:

We present Segment Anything Model (SAM) 3, a unified model that detects,
segments, and tracks objects in images and videos based on concept prompts,
which we define as either short noun phrases (e.g., “yellow school bus”), image
exemplars, or a combination of both. Promptable Concept Segmentation (PCS)
takes such prompts and returns segmentation masks and unique identities for all
matching object instances. To advance PCS, we build a scalable data engine that
produces a high-quality dataset with 4M unique concept labels, including hard
negatives, across images and videos. Our model consists of an image-level detector
and a memory-based video tracker that share a single backbone. Recognition and
localization are decoupled with a presence head, which boosts detection accuracy.
SAM 3 doubles the accuracy of existing systems in both image and video PCS,
and improves previous SAM capabilities on visual segmentation tasks. We open
source SAM 3 along with our new Segment Anything with Concepts (SA-Co)
benchmark for promptable concept segmentation.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 4183