Convolutional LSTMs for Cloud-Robust Segmentation of Remote Sensing Imagery

Marc Rußwurm, Marco Körner

Keywords: Convolutional LSTM, Cloud Filtering, Segmentation, Remote Sensing, Sentinel 2, Earth Observation, Recurrent Neural Networks, Vegetation Modelling, Land Cover Classification

TL;DR: We trained a ConvLSTM on vegetation classification with a sequence of satellite images and found that it learned to filter clouds without training on specific cloud labels.

Abstract: Clouds frequently cover the Earth's surface and pose an omnipresent challenge to optical Earth observation methods. The vast majority of remote sensing approaches either selectively choose single cloud-free observations or employ a pre-classification strategy to identify and mask cloudy pixels. We follow a different strategy and treat cloud coverage as noise that is inherent to the observed satellite data. In prior work, we directly employed a straightforward \emph{convolutional long short-term memory} network for vegetation classification without explicit cloud filtering and achieved state-of-the-art classification accuracies. In this work, we investigate this cloud-robustness further by visualizing internal cell activations and performing an ablation experiment on datasets of different cloud coverage. In the visualizations of network states, we identified some cells in which modulation and input gates closed on cloudy pixels. This indicates that the network has internalized a cloud-filtering mechanism without being specifically trained on cloud labels. Overall, our results question the necessity of sophisticated pre-processing pipelines for multi-temporal deep learning approaches.