Contrastive Learning for Domain Transfer in Cross-Corpus Emotion Recognition

Yufeng Yin, Liupei Lu, Yao Xiao, Zhi Xu, Kaijie Cai, Haonan Jiang, Jonathan Gratch, Mohammad Soleymani

Abstract: Automatic emotion recognition methods are sensitive to the variations across humans and datasets and their performance drops when evaluated across corpora. Domain adaptation (DA) techniques such as Domain-Adversarial Neural Network (DANN) can mitigate this problem. However, domain adaptation cannot guarantee to preserve local features necessary for emotion recognition while reducing domain discrepancies in global features. In this paper, we propose Face wArping emoTion rEcognition (FATE) to address this problem. Unlike the traditional DA models in which the base model is first trained with the source data and then fine-tuned with the source and target data, we reverse the training order. Specifically, we employ first-order facial animation warping to generate a synthetic dataset and utilize contrastive learning to pre-train the encoder. Then, we fine-tune the encoder and the classifier with the source data. After fine-tuning, the model achieves superior emotion recognition performance by preserving the subtle facial features. Our experiments on cross-domain emotion recognition with facial behaviors (Aff-Wild2, SEWA, and SEMAINE) indicate that the proposed FATE model substantially outperforms the domain adaptation models, suggesting that FATE has a better domain generalizability for emotion recognition.