FoSSIL: A Unified Framework for Continual Semantic Segmentation in 2D and 3D Domains

ICLR 2026 Conference Submission18706 Authors

Keywords: Incremental Learning, Few shot Learning, Domain shift, continual learning, domain-incremental learning, semi-supervised learning, semantic segmentation

Abstract:

Evolving visual environments challenge continual semantic segmentation by
introducing the complexities of class-incremental learning, domain-incremental
learning, limiting available annotations, and necessitating the use of unlabeled data.
In this work, we present the framework FoSSIL (Few-shot Semantic Segmentation
for Incremental Learning), which extensively benchmarks continual semantic
segmentation, spanning both 2D natural scenes and 3D medical volumes. Our
evaluation encompasses diverse and realistic settings, leveraging both labeled
(few-shot) and unlabeled data. Building on this benchmark, we introduce
guided noise injection to mitigate overfitting due to novel few-shot classes
from various domains. Furthermore, we leverage semi-supervised learning
for unlabeled data to augment few-shot novel classes. We propose a filtering
mechanism to remove highly confident but incorrectly predicted pseudo-labels,
further improving performance. Results across class-incremental, few-shot, and
domain-incremental scenarios with unlabeled data validate our strategies for
robust semantic segmentation in complex, evolving settings, highlighting both
the effectiveness and generality of our approach. Our findings illustrate that the
proposed framework forms a simple yet powerful recipe for continual semantic
segmentation in dynamic real-world environments. Our large-scale benchmarking
across natural 2D and medical 3D domains exposes key failure modes of existing
methods and offers a roadmap for building robust continual segmentation models.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 18706