A Rose by Any Other Name would not Smell as Sweet: Social Bias in Names Mistranslation

Sandra Camille Sandoval, Jieyu Zhao, Marine Carpuat, Hal Daumé III

Submission Type: Regular Long Paper

Submission Track: Ethics in NLP

Submission Track 2: Human-Centered NLP

Keywords: fairness and bias, NLP-related harms, name translation, evaluation

TL;DR: We develop an approach to measure disparities in the machine translation of first names across race/ethnicity and gender on translation accuracy and fluency; we find significant negative effects for Black and Hispanic female-associated names.

Abstract: We ask the question: Are there widespread disparities in machine translations of names across race/ethnicity, and gender? We hypothesize that the translation quality of names and surrounding context will be lower for names associated with US racial and ethnic minorities due to these systems’ tendencies to standardize language to predominant language patterns. We develop a dataset of names that are strongly demographically aligned and propose a translation evaluation procedure based on round-trip translation. We analyze the effect of name demographics on translation quality using generalized linear mixed effects models and find that the ability of translation systems to correctly translate female-associated names is significantly lower than male-associated names. This effect is particularly pronounced for female-associated names that are also associated with racial (Black) and ethnic (Hispanic) minorities. This disparity in translation quality between social groups for something as personal as someone's name has significant implications for people's professional, personal, and cultural identities, self-worth and ease of communication. Our findings suggest that more MT research is needed to improve the translation of names and to provide high-quality service for users regardless of gender, race, and ethnicity.

Submission Number: 1919