RegBN: Batch Normalization of Multimodal Data with Regularization

MORTEZA GHAHREMANI, Christian Wachinger

Keywords: Multimodal Data, Multimodality, Batch Normalization, Heterogeneous data, Regularization, Confounder, Confounding Effect Removal, Data Dependency

TL;DR: RegBN is a novel normalization method for multimodal data based on regularization that mitigates the confounding effects and the underlying dependencies among multimodal data, thereby enhancing the learning of multimodal representations.

Abstract: Recent years have witnessed a surge of interest in integrating high-dimensional data captured by multisource sensors, driven by the impressive success of neural networks in integrating multimodal data. However, the integration of heterogeneous multimodal data poses a significant challenge, as confounding effects and dependencies among such heterogeneous data sources introduce unwanted variability and bias, leading to suboptimal performance of multimodal models. Therefore, it becomes crucial to normalize the low- or high-level features extracted from data modalities before their fusion takes place. This paper introduces RegBN, a novel approach for multimodal Batch Normalization with REGularization. RegBN uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencies among different data sources. The proposed method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. We validate the effectiveness of RegBN on eight databases from five research areas, encompassing diverse modalities such as language, audio, image, video, depth, tabular, and 3D MRI. The proposed method demonstrates broad applicability across different architectures such as multilayer perceptrons, convolutional neural networks, and vision transformers, enabling effective normalization of both low- and high-level features in multimodal neural networks. RegBN is available at https://mogvision.github.io/RegBN.

Submission Number: 8404