Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning

Zuyao You

Track: Regular papers (within 8 pages excluding appendix)

Keywords: Segmentation, Large Multimodal Models

TL;DR: In this paper, we explore how to endow large language models (LLMs) with open-world segmentation capabilities using purely reinforcement learning without text supervision.

Abstract: We present Seg-R1, a preliminary exploration of using reinforcement learning (RL) to enhance the pixel-level understanding and reasoning capabilities of large multimodal models (LMMs). Starting with foreground segmentation tasks, specifically camouflaged object detection (COD) and salient object detection (SOD), our approach enables the LMM to generate point and bounding box prompts in the next-token fashion, which are then used to guide SAM2 in producing segmentation masks. We introduce Group Relative Policy Optimization (GRPO) into the segmentation domain, equipping the LMM with pixel-level comprehension through a carefully designed training strategy. Notably, Seg-R1 achieves remarkable performance with purely RL-based training, achieving .873 S-measure on COD10K without complex model modification. Moreover, we found that pure RL training demonstrates strong open-world generalization. Despite being trained solely on foreground segmentation image-mask pairs without text supervision, Seg-R1 achieves impressive zero-shot performance on referring segmentation and reasoning segmentation tasks, with 71.4 cIoU on RefCOCOg test and 56.7 gIoU on ReasonSeg test, outperforming models fully supervised on these datasets.

Confirmation: I have read and agree with the workshop's policy on behalf of myself and my co-authors.

Submission Number: 21