ZegOT: Zero-shot Segmentation Through Optimal Transport of Pixels to Text Prompts

Kwanyoung Kim, Yujin Oh, Jong Chul Ye

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Zero-shot semantic segmentation, Optimal transport, Prompt learning, Vision-language model

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We incorporate optimal transport theory into zero-shot semantic segmentation via multiple text-prompt learning

Abstract:

Recent success of large-scale Contrastive Language-Image Pre-training (CLIP) has led to great promise in zero-shot semantic segmentation by transferring text-image aligned knowledge to pixel-level classification.
However, existing methods usually require an additional image encoder or retraining/tuning the CLIP module.
Here, we propose a novel Zero-shot segmentation with Optimal Transport (ZegOT) method that matches multiple text prompts with frozen image embeddings through optimal transport. In particular, we introduce a novel Multiple Prompt Optimal Transport Solver (MPOT), which is designed to learn an optimal mapping between multiple text prompts and pixel embeddings of the frozen image encoder layers. This unique mapping method facilitates each of the multiple text prompts to effectively focus on distinct visual semantic attributes and diversify the learned knowledge to robustly cope with previously unseen categories.
Through extensive experiments on benchmark datasets, we show that our method achieves the state-of-the-art (SOTA) performance over existing Zero-shot Semantic Segmentation (ZS3) approaches.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4612