Towards Integrating Uncertainty for Domain-Agnostic Segmentation

Track: Main Track

Keywords: Segment Anything Models, Uncertainty Quantification, Laplace Approximation, Post-hoc Refinement

TL;DR: The paper explores lightweight post-hoc uncertainty measures to improve the Segment Anything Model's robustness to domain shifts, and introduces UncertSAM, a benchmark for testing segmentation robustness across varied challenges.

Abstract: Foundation models for segmentation such as the Segment Anything Model (SAM) family exhibit strong zero-shot performance, but remain vulnerable in shifted or limited-knowledge domains. This work investigates whether uncertainty quantification can mitigate such challenges and enhance model generalisability in a domain-agnostic manner. To this end, we (1) curate UncertSAM, a benchmark comprising eight datasets designed to stress-test SAM under challenging segmentation conditions including shadows, transparency, and camouflage; (2) evaluate a suite of lightweight, post-hoc uncertainty estimation methods; and (3) assess a preliminary uncertainty-guided prediction refinement step. Among evaluated approaches, a last-layer Laplace approximation yields uncertainty estimates that correlate well with segmentation errors, indicating a meaningful signal. While refinement benefits are preliminary, our findings underscore the potential of incorporating uncertainty into segmentation models to support robust, domain-agnostic performance.

Submission Number: 49