MatchMask: Mask-Centric Generative Data Augmentation for Label-Scarce Semantic Segmentation
Keywords: Semantic Segmentation, Generative Data Augmentation, Label-Scarce
Abstract: Current semantic segmentation models are very data-hungry and require massive costly pixel-wise human annotations. Generative data augmentation, which scales the train set using generative models, provides a potential remedy. However, existing text-centric methods struggle to generate complex in-distribution data due to the limitations of text descriptions. In this paper, we propose MatchMask, a novel mask-centric generative data augmentation approach tailored for label-scarce semantic segmentation. It leverages a few labeled semantic masks to generate diverse, realistic, and well-aligned image-mask training pairs for semantic segmentation models. Specifically, to adapt existing text-to-image models for semantic image synthesis, we first propose a Gradient Probe Method to investigate the role of each layer in the diffusion model. On this basis, we introduce a Layer-Timestep Adaptive Adapter (LT-Adapter) comprising layer-adaptive cross-attention fusion and time-adaptive LoRA scaling to enable efficient adaption for the critical layers. Meantime, we design a robust relative filtering principle to suppress incorrectly synthesized regions. Moreover, the proposed approach is extended to MatchMask++ in the semi-supervised setting to take advantage of additional unlabeled data. Experimental results on VOC, COCO and ADE20K demonstrate that MatchMask remarkably enhances the performance of segmentation models, surpassing prior data augmentation techniques in various benchmarks, \eg, 67.5\%$\rightarrow$74.3\% mIoU on VOC. Our code will be made publicly available.
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2851
Loading