Amanda: Adaptively Modality-Balanced Domain Adaptation for Multimodal Emotion Recognition

Anonymous

Abstract: This paper investigates unsupervised multimodal domain adaptation for multimodal emotion recognition, which is a solution for data scarcity yet remains under studied. Due to the varying distribution discrepancies of different modalities between source and target domains, the primary challenge lies in how to balance the domain alignment across modalities to guarantee they are all well aligned. To achieve this, we first develop our model based on the information bottleneck theory to learn optimal representation for each modality independently. Then, we align the domains via matching the label distributions and the representations. In order to balance the representation alignment, we propose to minimize a surrogate of the alignment losses, which is equivalent to adaptively adjusting the weights of the modalities throughout training, thus achieving balanced domain alignment across modalities. Overall, the proposed approach features \textbf{A}daptively \textbf{m}odality-bal\textbf{an}ced \textbf{d}omain \textbf{a}daptation, dubbed as \textbf{Amanda}, for multimodal emotion recognition. Extensive empirical results on commonly used benchmark datasets demonstrate that Amanda significantly outperforms competing approaches. The code is submitted as supplementary material, and the extracted features of the datasets will be made publicly available upon the publication of the paper.

Paper Type: long

Research Area: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English