RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything
Keywords: segment anything; real-time segmentation; multi-purpose model;
Abstract: Recent segmentation methods, which adopt large-scale data training and transformer architecture, aim to create one foundation model that can perform multiple tasks.
However, most of these methods rely on heavy encoder and decoder frameworks, hindering their performance in real-time scenarios.
To explore real-time segmentation, recent advancements primarily focus on semantic segmentation within specific environments, such as autonomous driving. However, they often overlook the generalization ability of these models across diverse scenarios.
Therefore, to fill this gap, this work explores a novel real-time segmentation setting called real-time multi-purpose segmentation.
It contains three fundamental sub-tasks: interactive segmentation, panoptic segmentation, and video instance segmentation.
Unlike previous methods, which use a specific design for each task, we aim to use only a single end-to-end model to accomplish all these tasks in real-time.
To meet real-time requirements and balance multi-task learning, we present a novel dynamic convolution-based method, Real-Time Multi-Purpose SAM (RMP-SAM).
It contains an efficient encoder and an efficient decoupled adapter to perform prompt-driven decoding.
Moreover, we further explore different training strategies and one new adapter design to boost co-training performance further.
We benchmark several strong baselines by extending existing works to support our multi-purpose segmentation.
Extensive experiments demonstrate that RMP-SAM is effective and generalizes well on proposed benchmarks and other specific semantic tasks.
Our implementation of RMP-SAM achieves the optimal balance between accuracy and speed for these tasks. The code is released at
\url{https://github.com/xushilin1/RAP-SAM}
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9393
Loading