Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation

Niccolò Avogaro, Thomas Frick, Mattia Rigotti, Andrea Bartezzaghi, Filip Janicki, Cristiano Malossi, Konrad Schindler, Roy Assaf

Keywords: Foundation Models, Prompting, Semantic Segmentation, Vision-Language Models, VLM, Training-free

Abstract:

Large Vision-Language Models (VLMs) are increasingly being regarded as foundation models that can be instructed to solve diverse tasks by prompting, without task-specific training.
We examine the seemingly obvious question: \emph{how to effectively prompt VLMs for semantic segmentation}.
To that end, we systematically evaluate the segmentation performance of several recent models guided by either text or visual prompts on the diverse MESS dataset collection.
We introduce a scalable prompting scheme, \emph{few-shot prompted semantic segmentation}, inspired by open-vocabulary segmentation and few-shot learning.
It turns out that even the most advanced VLMs lag far behind specialist models trained for a specific segmentation task, by about 30\% on average on the Intersection-over-Union metric.
Moreover, we find that text prompts and visual prompts are complementary: each one of the two modes fails on many examples that the other one can solve.
Our analysis suggests that being able to anticipate the most effective prompt modality can lead to a 11\% improvement in performance.
Motivated by our findings, we propose PromptMatcher, a remarkably simple baseline that combines both text and visual prompts, achieving state-of-the-art results for training-free semantic segmentation.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7189