Semantic-Centric Alignment for Zero-shot Panoptic and Semantic Segmentation

Jialei Chen, Qi Fan, Daisuke Deguchi, Chenkai ZHANG, Xu Zheng, Hiroshi Murase

Keywords: zero-shot segmentation, zero-shot learning, feature alignment, semantic segmentation, panoptic segmentation, semantic-centric

Abstract: Zero-shot segmentation has achieved great success by generating features from semantic embeddings to adapt the model to unseen classes. These semantic-generated features are typically aligned with the visual distribution of seen classes to improve generalization on extracted image features. However, this vision-centric alignment may easily overfit seen classes due to the lack of visual data for unseen classes. To address this issue, we propose a semantic-centric alignment method that aligns the generated features with the well-structured semantic distribution across all classes. First, we align the vision backbone features with CLIP tokens through Vision-to-CLIP alignment. This approach leverages CLIP’s visual-language matching capabilities to produce semantic-aligned backbone features. Then, we generate synthetic features from semantic embeddings for unseen classes, supervised by semantic-aligned visual features and CLIP semantic tokens for improving visual diversity while maintaining semantic consistency. Finally, we finetune the class projector through the semantic-aligned joint features to further adapt the model for unseen classes. Our semantic-centric alignment effectively enhances the model’s zero-shot generalization by constructing a unified and well-structured semantic-aligned feature space. Our method achieves SOTA performance in both zero-shot panoptic and semantic segmentation, and can directly segment unseen classes without fine-tuning.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3527