Excluding the Interference for Open-Vocabulary Semantic Segmentation

Shuai Shao, Shiyuan Zhao, Rui Xu, Yan Wang, Baodi Liu, Weifeng Liu, Yicong Zhou

Abstract: Open-vocabulary semantic segmentation (OVSS) is a hot research domain aimed at pixel-level categorization in dynamic environments, requiring the identification of both familiar categories and those known only by name but never visually encountered, offering significant practical value. Mainstream solutions integrate CLIP for category identification but often bias the model to misclassify novel categories as common ones (i.e., interference terms) due to inherent category imbalances within CLIP and exclusive reliance on known-class images for training. To address this issue, we introduce a novel approach named EXcluding the Interference Semantic SegmenTation Network (EXIST-Net), an extension of ELSE-Net, first presented at AAAI 2025. EXIST-Net transforms conventional single-step recognition into a nuanced two-stage process: initially filtering out interference terms to narrow the selection range, followed by enabling more precise identification of the sample’s specific category. In implementation, EXIST-Net consists of four blocks: (1) Mask Proposal Network (MPN) generates class-agnostic masks. (2) Mask Forward Classifier (MFC) assesses the inclusion probability (the likelihood that a mask belongs to a category). (3) Mask Reverse Classifier (MRC) is the cornerstone to implement the “Excluding the Interference” concept. It calculates high-quality exclusion probabilities (the likelihood that a mask does not belong to a specific category). (4) Probability Corrector (PCor) leverages exclusion probabilities to adjust inclusion probabilities, thereby improving the accuracy of semantic segmentation. Moreover, the MRC block is model-agnostic and entails low consumption, making it compatible with a wide range of mainstream approaches. Experimental results on five benchmark datasets validate the effectiveness of EXIST-Net and demonstrate the model-agnostic functionality and low resource usage of the MRC block.

External IDs:doi:10.1109/tcsvt.2026.3650803