MMANet: Margin-Aware Distillation and Modality-Aware Regularization for Incomplete Multimodal Learning

Shicai Wei, Chunbo Luo, Yang Luo

Abstract:

Multimodal learning has shown great potentials in numerous
scenes and attracts increasing interest recently.
However, it often encounters the problem of missing modality
data and thus suffers severe performance degradation
in practice. To this end, we propose a general framework
called MMANet to assist incomplete multimodal learning.
It consists of three components: the deployment network
used for inference, the teacher network transferring
comprehensive multimodal information to the deployment
network, and the regularization network guiding the deployment
network to balance weak modality combinations.
Specifically, we propose a novel margin-aware distillation
(MAD) to assist the information transfer by weighing
the sample contribution with the classification uncertainty.
This encourages the deployment network to focus
on the samples near decision boundaries and acquire the
refined inter-class margin. Besides, we design a modalityaware
regularization (MAR) algorithm to mine the weak
modality combinations and guide the regularization network
to calculate prediction loss for them. This forces
the deployment network to improve its representation ability
for the weak modality combinations adaptively. Finally,
extensive experiments on multimodal classification
and segmentation tasks demonstrate that our MMANet outperforms
the state-of-the-art significantly.