Spatial-Temporal-Spectral Unified Modeling for Remote Sensing Dense Prediction

Sijie Zhao, Feng Liu, Enzhuo Zhang, Yiqing Guo, Pengfeng Xiao, Xueliang Zhang, LEI BAI, Hao Chen

Keywords: Remote Sensing, Dense Prediction, Data Unification, Task Unification, Category Unification

TL;DR: Proposed a unified remote sensing dense prediction model that adapt to arbitrary input and output, multiple tasks and semantic classes.

Abstract: The proliferation of multi-source remote sensing data has propelled the development of deep learning for dense prediction, yet significant challenges in data and task unification persist. Current deep learning architectures for remote sensing are fundamentally rigid. They are engineered for fixed input-output configurations, restricting their adaptability to the heterogeneous spatial, temporal, and spectral dimensions inherent in real-world data. Furthermore, these models fail to leverage the intrinsic correlations across different remote sensing dense prediction tasks, necessitating the development of distinct models or task-specific decoders. This paradigm is also limited to a fixed set of output semantic classes that must be learned during training, where any change to the classes requires costly retraining. To overcome these limitations, we introduce the Spatial-Temporal-Spectral Unified Network (STSUN) for unified modeling. STSUN can adapt to input and output data with arbitrary spatial sizes, temporal lengths, and spectral bands by leveraging their metadata for a unified representation. Moreover, STSUN unifies disparate dense prediction tasks within a single architecture by conditioning the model on trainable task embeddings. STSUN enables flexible prediction across multiple sets of semantic categories by integrating trainable category embeddings as metadata. Extensive experiments on multiple datasets with diverse Spatial-Temporal-Spectral configurations in multiple scenarios demonstrate that a single STSUN model effectively adapts to heterogeneous inputs and outputs, unifying various dense prediction tasks and diverse semantic class predictions.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Submission Number: 9036