Nighttime Semantic Segmentation with Unsupervised Learning and Cross Attention

Jian Cheng, Yang-Feng Hu, Yu Dai, Xue Qiao, Li Yao, Jun-Yan Yang

Abstract: In recent years, semantic segmentation has shown very good performance in daytime scenes. But in nighttime scenes, semantic segmentation greatly reduces its accuracy. Due to the lack of large-scale nighttime semantic segmentation datasets, it is difficult to directly train segmentation models for nighttime scenes. Therefore, it becomes important to adapt the daytime scene segmentation model to the nighttime scene without directly using the nighttime scene segmentation dataset. In this paper, we propose a framework based on unsupervised learning and cross attention. The proposed method fuses supervised daytime scenes and unsupervised nighttime scenes, the supervision information in the daytime scene and the texture information specific to the nighttime scene are fully utilized, and the model is adapted to both the daytime scene and the nighttime scene. Consistency regulation is used to make segmentation model adapt to the complex and changeable night scene texture and illumination. In view of the coarse correspondence of static objects between day and night image pairs in the Dark Zurich dataset, cross attention is proposed to make the model pay more attention to the parts of the night scene which are similar to the daytime scene. Extensive experiments on Dark Zurich and Nighttime Driving datasets show that our method obtains better performance in nighttime semantic segmentation.