Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation

Guangrui Li, Guoliang Kang, Xiaohan Wang, Yunchao Wei, Yi Yang

Abstract:

This paper considers the synthetic-to-real adaptation of
point cloud semantic segmentation, which aims to segment
the real-world point clouds with only synthetic labels available. Contrary to synthetic data which is integral and
clean, point clouds collected by real-world sensors typically contain unexpected and irregular noise because the
sensors may be impacted by various environmental conditions. Consequently, the model trained on ideal synthetic
data may fail to achieve satisfactory segmentation results
on real data. Influenced by such noise, previous adversarial training methods, which are conventional for 2D adaptation tasks, become less effective. In this paper, we aim to
mitigate the domain gap caused by target noise via learning to mask the source points during the adaptation procedure. To this end, we design a novel learnable masking
module, which takes source features and 3D coordinates as
inputs. We incorporate Gumbel-Softmax operation into the
masking module so that it can generate binary masks and be
trained end-to-end via gradient back-propagation. With the
help of adversarial training, the masking module can learn
to generate source masks to mimic the pattern of irregular
target noise, thereby narrowing the domain gap. We name
our method “Adversarial Masking” as adversarial training
and learnable masking module depend on each other and
cooperate with each other to mitigate the domain gap. Experiments on two synthetic-to-real adaptation benchmarks
verify the effectiveness of the proposed method