Decoupling Features for Remote Sensing Missing Modality Learning

Yiheng Zhou, Ailong Ma, Junjue Wang, Zihang Chen, Yanfei Zhong

Abstract: Missing modality learning aims to enhance the robustness of the model when certain modalities are missing during the testing phase, by transferring knowledge between different modalities during the training stage. Existing works directly reduce the distance between different modalities feature through auxiliary tasks. While simple, this approach shows limited capability in some complex scenarios. It disrupts the distribution of modality-specific features, leading to a decline in modality-discriminative capability. Therefore, we prosed the DFNet. DFNet is designed to decouple the feature into modality-invariant and modality-variant features, by whitening transformation loss. Only the content features of different modalities are brought closer through auxiliary tasks. Experiments are conducted on MSAW datasets, with results indicating that our model outperforms competing methods.

External IDs:dblp:conf/igarss/ZhouMWCZ24