EvoProto: Evolving Prototypes with Class Similarity for Weakly Incremental Segmentation

Kai Fang, Zhonggai Wang, Guangyu Gao, Jianbo Jiao, Chi Harold Liu, Yunchao Wei

Keywords: weakly supervised learning, class overwriting, class similarity, incremental semantic segmentation, prototype learning

Abstract: Weakly Supervised Incremental Learning for Semantic Segmentation (WILSS) seeks to segment new classes using only image-level labels, without access to old class data, which challenges the stability-plasticity balance. The absence of pixel-level annotations for new classes and historical data for old classes often leads to class overwriting, where predictions for new classes misclassify or override regions belonging to semantically similar previously learned classes. We observe that such overwriting frequently arises from class confusion, where visually similar classes are entangled due to weak supervision and limited feature discrimination. To address this, we propose EvoProto, a framework that explicitly models and mitigates class confusion through the dynamic evolution of trainable class prototypes. We begin by introducing a confusion score that quantifies semantic similarity between new and old classes. Computed from CAM-derived predictions after a warm-up phase, this score is transformed into adaptive weights that guide both contrastive prototype learning and prototype-level knowledge distillation, thereby reinforcing inter-class separability during continual updates. Besides, each class in EvoProto is associated with a learnable prototype vector, which is continuously updated during training to better capture class-specific semantics and improve discriminability under weak supervision. Additionally, to counter the degradation in classification capability and the resulting pseudo-label noise during incremental steps in weak supervision, we propose a CAM Channel Selection mechanism that emphasizes confident and consistent activations as more reliable supervision. Extensive experiments on Pascal VOC and COCO benchmarks demonstrate that EvoProto effectively alleviates class overwriting and achieves state-of-the-art performance under various incremental scenarios. The code will be made publicly available.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 5684