Open-Set Domain Generalization for Semantic Segmentation

Seun-An Choe, Keonhee Park, Jinwoo Choi, Gyeong-Moon Park

Keywords: Semantic Segmentation, Domain Generalization

TL;DR: Unknown-aware segmentation in unseen domains

Abstract: Open-Set domain generalization for semantic segmentation (OSDG-SS) aims to segment known classes and identify unknown categories in target domains that are entirely unseen during training. While recent domain generalization methods perform well under the closed-set assumption, they struggle in open-set settings by misclassifying unknown objects as one of the known classes. To address this challenge, we propose a unified framework that explicitly models unknowns and improves robustness to both semantic and visual domain shifts. First, to provide supervision for unknown regions, we generate realistic unknown objects using Stable Diffusion and insert them into source images, allowing the model to learn unknown-aware representations via segmentation head expansion. However, since synthetic unknowns may not reflect the true distribution of unknowns in target domains, we introduce a meta-learning strategy that partitions the unknown set into meta-train and meta-test subsets, guiding the model to generalize across unseen unknown categories through entropy-based rejection and subdomain shifts. Finally, to reduce confusion between unknowns and visually similar known classes, we optimize the decision boundaries in feature space by enforcing compactness for known classes and expanding the unknown using Mixup-based hard negative synthesis. Extensive experiments across multiple benchmarks demonstrate that our framework significantly improves in the OSDG-SS setting.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 23772