Galileo: Learning Global & Local Features of Many Remote Sensing Modalities

Gabriel Tseng, Anthony Fuller, Marlena Reil, Henry Herzog, Patrick Beukema, Favyen Bastani, James R Green, Evan Shelhamer, Hannah Kerner, David Rolnick

TL;DR: Flexible, multimodal pretrained remote sensing models which can model phenomena at different scales

Abstract: We introduce a highly multimodal transformer to represent many remote sensing modalities - multispectral optical, synthetic aperture radar, elevation, weather, pseudo-labels, and more - across space and time. These inputs are useful for diverse remote sensing tasks, such as crop mapping and flood detection. However, learning shared representations of remote sensing data is challenging, given the diversity of relevant data modalities, and because objects of interest vary massively in scale, from small boats (1-2 pixels and fast) to glaciers (thousands of pixels and slow). We present a novel self-supervised learning algorithm that extracts multi-scale features across a flexible set of input modalities through masked modeling. Our dual global and local contrastive losses differ in their targets (deep representations vs. shallow input projections) and masking strategies (structured vs. not). Our Galileo is a single generalist model that outperforms SoTA specialist models for satellite images and pixel time series across eleven benchmarks and multiple tasks.

Lay Summary:

We capture a lot of information about our planet from “remote sensing data” (satellite observations, topographic maps, and more) but we know less than you might think. Analyzing remote sensing data with machine learning can help us better understand our changing planet.

We present a machine learning model — which we call Galileo — that can help summarize remote sensing data. This means that with minimal further processing, its summaries can help make predictions and maps, like of floods or agricultural fields.

We achieve this by giving Galileo an incomplete set of data for a time and place, and having it reconstruct what we removed. By being careful about exactly what we ask Galileo to reconstruct, we can make sure Galileo’s summaries take into account big and slow things (like glaciers) as well as small and fast things (like fishing boats).

Galileo is uniquely relevant to remote sensing in practice by its modeling of data across space, time, and a variety of data types (e.g. optical data from satellites, topographic maps, weather data and more). We test Galileo on 15 diverse tasks against 11 other methods: Galileo performs best with a single general model. This makes it immediately useful in many existing applications.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/nasaharvest/galileo

Primary Area: Applications->Everything Else

Keywords: remote sensing, ai for good, land cover mapping, satellite data, agriculture

Flagged For Ethics Review: true

Submission Number: 4513