ERL-MR: Harnessing the Power of Euler Feature Representations for Balanced Multi-modal Learning

Weixiang Han, Chengjun Cai, Guo Yu, Jialiang Peng

Abstract: Multi-modal learning leverages data from diverse perceptual media to obtain enriched representations, thereby empowering machine learning models to complete more complex tasks. However, recent research results indicate that multi-modal learning still suffers from “modality imbalance”: Certain modalities' contributions are suppressed by dominant ones, consequently constraining the overall performance enhancement of multimodal learning. To tackle this issue, current approaches attempt to mitigate modality competition in various ways, but their effectiveness is still limited. To this end, we propose an Euler Representation Learning-based Modality Rebalance (ERL-MR) strategy, which reshapes the underlying competitive relationships between modalities into mutually reinforcing win-win situations while maintaining stable feature optimization directions. Specifically, ERL-MR employs Euler's formula to map original features to complex space, constructing cooperatively enhanced non-redundant features for each modality, which helps reverse the situation of modality competition. Moreover, to counteract the performance degradation resulting from optimization drift among modalities, we propose a Multi-Modal Constrained (MMC) loss based on cosine similarity of complex feature phase and cross-entropy loss of individual modalities, guiding the optimization direction of the fusion network. Extensive experiments conducted on four multi-modal multimedia datasets and two task-specific multi-modal multimedia datasets demonstrate the superiority of our ERL-MR strategy over state-of-the-art baselines, achieving modality rebalancing and further performance improvements.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Content] Multimodal Fusion

Relevance To Conference: This paper is dedicated to addressing the issue of modality imbalance in multimodal learning, aligning with the thematic focus outlined in the call for papers of ACM MM. Specifically, we introduce the ERL-MR strategy as a novel approach to mitigate modality imbalance in multimodal learning settings. ERL-MR stands out as a potent solution to tackle the challenges posed by modality imbalance, leading to substantial enhancements in model performance. These results underscore the promising potential of ERL-MR strategies in advancing robust multimodal learning for real-world application scenarios.

Submission Number: 3046