Open Domain Generalized Semantic Segmentation
Keywords: domain generalization, out-of-distribution, semantic segmentation
Abstract: Conventional domain generalized semantic segmentation (DGS) aims to learn a generalized segmentation model for unknown target domains by utilizing only the labeled source domain under distribution shifts. Typically, it assumes that the category sets are consistent across domains. However, in real-world applications, it is not always possible to find labeled source data that precisely cover the category sets of the target domains. Therefore, in this paper, we study a more practical and challenging problem of open domain generalized semantic segmentation (ODGS), which is a non-trivial task as it involves both distribution shifts and category set shifts (i.e., open categories in unknown target domains). To tackle these shifts, we introduce three Semantic-oriented components which together form a simple yet effective CLIP-adapted framework called SoCLIP. (1) Semantic Definition Prompt mitigates the visual-language ambiguity by incorporating category definitions into the text prompt template. (2) Semantic Prompt Tuning first queries a powerful large language model, e.g., ChatGPT, to obtain domain-agnostic semantic cues, and then project and tune these cue prompts to assist the fixed image encoder in exploring domain-invariant pixel-level knowledge. (3) Semantic-preserved Randomization leverages the Fourier Transform to randomize the source domain with semantic preservation, preventing overfitting to the training domain. With the integration of these components, SoCLIP significantly outperforms state-of-the-art methods in both open-set and closed-set (i.e., conventional) settings.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4614
Loading