Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization

Zhengqi Gao, Sucheng Ren, Zihui Xue, Siting Li, Hang Zhao

Keywords: Jacobian Regularization, Robust Multimodal Learning

Abstract: Multimodal fusion emerges as an appealing technique to improve model performances on many tasks. Nevertheless, the robustness of such fusion methods is rarely involved in the present literature. In this paper, we are the first to propose a training-free robust late-fusion method by exploiting conditional independence assumption and Jacobian regularization. Our key is to minimize the Frobenius norm of a Jacobian matrix, where the resulting optimization problem is relaxed to a tractable Sylvester equation. Furthermore, we provide a theoretical error bound of our method and some insights about the function of the extra modality. Several numerical experiments on AV-MNIST, RAVDESS, and VGGsound demonstrate the efficacy of our method under both adversarial attacks and random corruptions.

One-sentence Summary: We propose a training-free robust multimodal learning method via sample-wise Jacobian regularization.