When Tone and Words Disagree: Towards Robust Speech Emotion Recognition under Acoustic-Semantic Conflict

ACL ARR 2026 January Submission3361 Authors

Keywords: speech emotion recognition, acoustic-semantic conflict

Abstract: Speech Emotion Recognition (SER) systems often assume congruence between vocal emotion and lexical semantics. However, in real-world interactions, acoustic-semantic conflict is common yet overlooked, where the emotion conveyed by tone contradicts the literal meaning of spoken words. We show that state-of-the-art SER models, including ASR-based, self-supervised learning (SSL) approaches and Audio Language Models (ALMs), suffer performance degradation under such conflicts due to semantic bias or entangled acoustic–semantic representations. To address this, we propose the **Fusion Acoustic-Semantic (FAS)** framework, which explicitly disentangles acoustic and semantic pathways and bridges them through a lightweight, query-based attention module. To enable systematic evaluation, we introduce the **Conflict in Acoustic-Semantic Emotion(CASE)**, the first dataset dominated by clear and interpretable acoustic-semantic conflicts in varied scenarios. Extensive experiments demonstrate that FAS consistently outperforms existing methods in both in-domain and zero-shot settings. Notably, on the CASE benchmark, conventional SER models fail dramatically, while FAS sets a new SOTA with 59.38\% accuracy. Our code and datasets is available at https://anonymous.4open.science/r/FAS-Anonymous.

Paper Type: Long

Research Area: Speech Processing and Spoken Language Understanding

Research Area Keywords: spoken language understanding, speech technologies

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources, Data analysis

Languages Studied: English, Chinese, German, Italian

Submission Number: 3361