ReGRAF: Training free Prompt Refinement via Gradient Flow for Segmentation

Minjae Lee, Sungwoo Hur, Soojin Hwang, Won Hwa Kim

Keywords: Visual Foundation model, Segmentation, Computer Vision, Prompt engineering

Abstract:

Visual Foundation Models (VFMs) such as the Segment Anything Model (SAM) have significantly advanced segmentation, object detection, and image classification tasks.
However, SAM and its fine-tuned variants necessitate substantial manual effort for prompt generation and additional training for specific applications.
Recent methods have addressed these limitations by integrating SAM into one-shot and few-shot segmentation, enabling auto-prompting through semantic alignment between query and support images.
Despite these advancements, they still generate inadequate prompts that degrade segmentation quality.
To tackle this limitation, we introduce ReGRAF (Refinement via GRAdient Flow), a training-free method that refines prompts through gradient flow derived from SAM's mask decoder. ReGRAF seamlessly integrates into SAM-based auto-prompting frameworks and is theoretically proven to refine segmentation masks with high efficiency and precision. Extensive evaluations demonstrate that ReGRAF consistently improves segmentation quality across various benchmarks, effectively mitigating false positives without requiring additional training or architectural modifications.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4484