Balanced Multimodal Learning: An Unidirectional Dynamic Interaction Perspective

Shijie Wang, Li Zhang, Xinyan Liang, Yuhua Qian, Shen Hu

Keywords: Multimoadal learning

Abstract:

Multimodal learning typically utilizes multimodal joint loss to integrate different modalities and enhance model performance. However, this joint learning strategy can induce modality imbalance, where strong modalities overwhelm weaker ones and limit exploitation of individual information from each modality and the inter‑modality interaction information. Existing strategies such as dynamic loss weighting, auxiliary objectives and gradient modulation mitigate modality imbalance based on joint loss. These methods remain fundamentally reactive, detecting and correcting imbalance after it arises, while leaving the competitive nature of the joint loss untouched. This limitation drives us to explore an alternative approach that avoids reliance on the joint loss, aiming to foster more effective modality interactions and to better exploit both per-modality information and inter-modality complementarity. In this paper, we introduce Unidirectional Dynamic Interaction (UDI), a proactive sequential training strategy that replaces conventional joint optimization. UDI first trains the anchor modality to convergence, then uses its learned representations to guide the other modality via unsupervised loss. Furthermore, the dynamic adjustment of modality interactions allows the model to adapt to the task at hand, ensuring that each modality contributes optimally.
By decoupling modality optimization and enabling directed information flow, UDI prevents domination by any single modality and fosters effective cross‐modal feature learning. Our experimental results demonstrate that UDI outperforms existing methods in handling modality imbalance, leading to performance improvement in multimodal learning tasks. (The code will be published.)

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 4026