Mitigating Non-Uniform Forgetting Dynamics for Class-incremental Semantic Segmentation

Abstract:

Class-Incremental Semantic Segmentation (CISS) aims to learn newly introduced classes sequentially while preserving performance on previously learned ones. Most existing methods mitigate catastrophic forgetting through pseudo labels or regularization, but largely assume that forgetting evolves uniformly across old classes. In this paper, we reveal and characterize a Non-Uniform Forgetting(NUF) phenomenon in CISS, where different old classes exhibit markedly different forgetting trajectories in terms of degradation severity, onset time, and temporal pattern. Our analysis further shows that NUF is closely related to semantic complexity, semantic overlap, and the inherent old--new supervision imbalance of CISS.
To address this problem, we propose a pseudo-labels-assisted framework with two complementary components. The first, Imbalance-Aware Gradient Defence (IGD), alleviates optimization bias through pixel-wise gradient-aware reweighting and channel-wise balancing, while a background-suppression term further reduces spurious foreground activations. The second,Representation Drift Suppressor(RDS), improves representation stability by enhancing inter-class separability with prototype-based contrastive learning and preserving old semantics through selective decoder-level distillation. By jointly combining IGD and RDS, the proposed framework effectively mitigates heterogeneous forgetting and yields more balanced incremental segmentation performance.
Extensive experiments on PASCAL VOC and ADE20K under multiple incremental protocols demonstrate that the proposed method consistently improves old-class retention and overall incremental performance, outperforming state-of-the-art CISS approaches.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Xuming_He3

Submission Number: 8208