CGFNet: cross-guided fusion network for RGB-thermal semantic segmentation

Yanping Fu, Qiaoqiao Chen, Haifeng Zhao

Abstract: Semantic segmentation is a basic task in computer vision, which is widely used in various fields such as autonomous driving, detection, augmented reality and so on. Recent advances in deep learning have achieved commendable results in the semantic segmentation for visible (RGB) images. However, the performance of these methods will decline precipitously in dark or visually degraded environments. To overcome this problem, thermal images are introduced into semantic segmentation tasks because they can be captured in all-weather. To make full use of the two modal information, we propose a novel cross-guided fusion attention network for RGB-T semantic segmentation, which uses an attention mechanism to extract the weights of two modalities and guide each other. Then we extract the global information and add it to the decoding process. In addition, we propose a dual decoder to decode the features of different modalities. The two decoders will predict three prediction maps. We use these three prediction maps to train our network jointly. We conducted experiments on two public datasets, and the experimental results demonstrate the effectiveness of the proposed method.