Semantically-Aware Contrastive Learning for multispectral remote sensingimages

Leandro Stival, Ricardo da Silva Torres, Helio Pedrini

Abstract: Satellites continuously capture vast amounts of data daily, including multispectral remote sensing images(MSRSI), which facilitate the analysis of planetary processes and changes. New machine-learning techniquesare employed to develop models to identify regions with significant changes, predict land-use conditions, andsegment areas of interest. However, these methods often require large volumes of labeled data for effectivetraining, limiting the utilization of captured data in practice. According to current literature, self-supervisedlearning (SSL) can be effectively applied to learn how to represent MSRSI. This work introduces SemanticallyAware Contrastive Learning (SACo+), a novel method for training a model using SSL for MSRSI. Relevantknown band combinations are utilized to extract semantic information from the MSRSI and texture-basedrepresentations, serving as anchors for constructing a feature space. This approach is resilient against changesand yields semantically informative results using contrastive techniques based on sample visual properties,their categories, and their changes over time. This enables training the model using classic SSL contrastiveframeworks, such as MoCo and its remote sensing version, SeCo, while also leveraging intrinsic semanticinformation. SACo+ generates features for each semantic group (band combination), highlighting regions inthe images (such as vegetation, urban areas, and water bodies), and explores texture properties encoded basedon Local Binary Pattern (LBP). To demonstrate the efficacy of our approach, we trained ResNet models withMSRSI using the semantic band combinations in SSL frameworks. Subsequently, we compared these models onthree distinct tasks: land cover classification task using the EuroSAT dataset, change detection using the OSCDdataset, and semantic segmentation using the PASTIS and GID datasets. Our results demonstrate that leveragingsemantic and texture features enhances the quality of the feature space, leading to improved performance inall benchmark tasks. The model implementation and weights are available at https://github.com/lstival/SACo— As of Jan. 2025.