Emotion Recognition in Conversation Based on the Fine-grained Multidimensional Emotion Representation Learning

ACL ARR 2025 February Submission5324 Authors

Abstract: Traditional emotion recognition in conversation (ERC) studies usually are designed to predict a fixed set of predetermined emotion categories. This limited supervision diminishes the expressive power of the data, resulting in failing to capture the complexity of human emotions in conversation. Learning from a well-designed fine-grained representation of emotions offers a promising alternative that utilizes a wider range of supervision. In this paper, the proposed Fine-grained Multidimensional Emotion Representation Learning (FMERL) framework integrates multitask learning and contrastive learning, and extends the emotion representation of valence, arousal and dominance (VAD) from psychological field to both continuous and discrete forms. Firstly, the emotion features from text, audio and visual modalities are extracted. Then, the multimodal features are fused by a transformer-based model. The multitask learning contains three feedforward networks (FFNs), the valence net, arousal net, and dominance net, for learning the continuous fine-grained emotion representations from the fused multimodal features. The contrastive learning aligns fused multimodal features with the discrete fine-grained emotion representations derived through prompt engineering applied to a large language model. The transferable ability of contrastive learning enables FMERL to map the semantic information of emotion representation and fused multimodal features into a shared embedding space, thereby understanding their semantic relationships and enabling zero-shot learning. Experimental results on the IEMOCAP and MELD datasets have shown that FMERL achieves state-of-the-art performance in emotion recognition and implements zero-shot learning in the field of ERC.

Paper Type: Long

Research Area: Computational Social Science and Cultural Analytics

Research Area Keywords: emotion detection and analysis

Languages Studied: English

Submission Number: 5324