Changing Modalities by Cross-Band Transfer, Addition, and Peeking

Tim G. Zhou, Anthony Fuller, Geoff Pleiss, Evan Shelhamer

Abstract: Machine learning models for remote sensing typically assume a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy a model on a substitution, superset, or subset of modalities given data availability or practical constraints. We formulate the setting of changing modalities and identify three main scenarios: Modality Transfer, Addition, and Peeking. We propose Delulu-Net, an architecture with modular components that adapts to changes in modalities. Delulu-Net learns a multi-modal model from a unimodal teacher and unlabeled multimodal data, providing a practical alternative to re-labeling and re-training.

Submission Number: 30