SAM 3: Segment Anything with Concepts

ICLR 2026 Conference Submission4183 Authors

Keywords: foundation models, open vocabulary segmentation, semantic instance segmentation, object tracking

TL;DR: We present a strong model and challenging benchmark to advance open-vocabulary concept segmentation in images and videos.

Abstract:

We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short noun phrases (e.g., “yellow school bus”), image exemplars, or a combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching
object instances. To advance PCS, we build a scalable data engine that produces a high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of a vision backbone shared between an image-level detector and a memory-based video tracker. Recognition and localization are decoupled with a presence head, which significantly boosts detection accuracy. SAM 3 delivers a 2x gain over existing systems in both image and video PCS, and improves previous SAM capabilities in interactive visual segmentation tasks. We open source SAM 3 along with our new Segment Anything with Concepts (SA-Co) benchmark.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 4183