CCANet: Cross-Modality Comprehensive Feature Aggregation Network for Indoor Scene Semantic Segmentation

Zihao Zhang, Yale Yang, Huifang Hou, Fanman Meng, Fan Zhang, Kangzhan Xie, Chunsheng Zhuang

Abstract: The semantic segmentation of indoor scenes based on RGB and depth information has been a persistent and enduring research topic. However, how to fully utilize the complementarity of multimodal features and achieve efficient fusion remains a challenging research topic. To address this challenge, we proposed an innovative cross-modal comprehensive feature aggregation network (CCANet) to achieve high-precision semantic segmentation of indoor scenes. In this method, we first propose a bidirectional cross-modality feature rectification (BCFR) module to complement each other and remove noise in both channel and spatial correlations. After that, the adaptive criss-cross attention fusion (CAF) module is designed to realize multistage deep multimodal feature fusion. Finally, a multisupervision strategy is applied to accurately learn additional details of the target, guiding the gradual refinement of segmentation maps. By conducting thorough experiments on two openly accessible datasets of indoor scenes, the results demonstrate that CCANet exhibits outstanding performance and robustness in aggregating RGB and depth features.