AoP-SAM: Automation of Prompts for Efficient Segmentation
Keywords: SAM; Foundation Model; Prompt Tuning; Segmentation; Computer Vision; Automation
TL;DR: We propose AoP-SAM, a novel approach automatically generate essential prompts for accurate segmentation, eliminating the need for manual prompt provision.
Abstract: The Segment Anything Model (SAM) is a powerful foundation model for image segmentation, showing robust zero-shot generalization through prompt engineering.
However, relying on manual prompts is impractical for real-world applications, particularly in scenarios where rapid prompt provision and resource efficiency are crucial.
In this paper, we propose the Automation of Prompts for SAM (AoP-SAM), a novel approach that learns to generate essential prompts in optimal locations automatically. AoP-SAM enhances SAM’s efficiency and usability by eliminating manual input, making it better suited for real-world segmentation tasks.
Our approach employs a lightweight yet efficient Prompt Predictor model that detects key entities across images and identifies the optimal regions for placing prompt candidates. This method leverages SAM’s image embeddings, preserving its zero-shot generalization capabilities without requiring fine-tuning.
Additionally, we introduce a test-time instance-level Adaptive Sampling and Filtering mechanism that generates prompts in a coarse-to-fine manner. This significantly enhances both prompt and mask generation efficiency by reducing computational overhead and minimizing redundant mask refinements.
Evaluations of three datasets demonstrate that AoP-SAM substantially improves both prompt generation efficiency and mask generation accuracy, making SAM more effective for automated segmentation tasks.
Submission Number: 65
Loading