Causality-Guided Stepwise Intervention and Reweighting for Remote Sensing Image Semantic Segmentation

Shuting Shi, Baohong Li, Laifu Zhang, Kun Kuang, Sensen Wu, Tian Feng, Yiming Yan, Zhenhong Du

Abstract: Semantic segmentation is one of the most significant tasks in remote sensing (RS) image interpretation, which focuses on learning global and local information to infer the semantic label of each pixel. Previous studies devise encoder-decoder structured deep learning (DL) models to extract global and local features from RS images with the help of pretraining knowledge to predict semantic labels. However, due to the common heterogeneity between the data for pretraining and the data to be semantically segmented, these models fail to learn general features appropriate to RS datasets. In this article, we propose a novel formulation of the above problem from a causal perspective, where the learned features from pretrained models result from causality and spurious correlations, and only the former carries general information that remains invariant regardless of the exact task and dataset. Based on the above formulation, we propose stepwise intervention and reweighting (SIR). It can reduce the confounding bias introduced by the pretraining knowledge and improve the model’s ability to learn general features, making semantic segmentation of RS images benefit more from pretraining. Besides, we conduct a detailed theoretical analysis of our methods and conduct extensive experiments on two widely used public RS datasets. Experimental results demonstrate that applying SIR to encoder-decoder semantic segmentation models achieves performance improvements, proving the effectiveness and application values of the proposed method.