EMOS: A Comprehensive Evaluation Framework for Emotion Preservation in Machine Translation

ACL ARR 2025 May Submission3797 Authors

Abstract: Contemporary machine translation systems excel at preserving semantic content but inadequately address emotional dimensions critical for cross-cultural communication. We introduce EMOS (Emotion Preservation Score), a theoretically-grounded evaluation framework that transcends traditional sentiment analysis through multidimensional assessment of emotional fidelity. EMOS integrates three complementary metrics: Vector Similarity Score (VSS), Label Match Rate (LMR), and Emotional Diversity Ratio (EDR), weighted to capture distributional similarity, categorical preservation, and emotional complexity maintenance. Through empirical validation on classical Chinese literature translated by DeepL, Google Translate, and GPT-4o, we demonstrate that EMOS effectively captures emotion preservation quality invisible to traditional metrics. Results show that while all systems achieve good emotional fidelity (EMOS > 0.75), GPT-4o exhibits superior performance (0.780) compared to DeepL and Google Translate (both 0.757), particularly for culturally-embedded emotional expressions.

Paper Type: Long

Research Area: Machine Translation

Research Area Keywords: Machine Translation, automatic evaluation, Sentiment Analysis

Contribution Types: Approaches to low-resource settings, Data resources, Data analysis

Languages Studied: Chinese, English

Submission Number: 3797