Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis
Submission Type: Regular Long Paper
Submission Track: Ethics in NLP
Submission Track 2: Multilinguality and Linguistic Diversity
Keywords: bias, fairness, multilingual, sentiment analysis, counterfactual, contrastive
TL;DR: We do the first test for whether cross-lingual transfer can increase/decrease gender and racial biases in sentiment analysis, as compared to monolingual systems. We find that it can.
Abstract: Sentiment analysis (SA) systems are widely deployed in many of the world's languages, and there is well-documented evidence of demographic bias in these systems. In languages beyond English, scarcer training data is often supplemented with transfer learning using pre-trained models, including multilingual models trained on other languages. In some cases, even supervision data comes from other languages. Does cross-lingual transfer also import new biases? To answer this question, we use counterfactual evaluation to test whether gender or racial biases are imported when using cross-lingual transfer, compared to a monolingual transfer setting. Across five languages, we find that systems using cross-lingual transfer usually become more biased than their monolingual counterparts. We also find racial biases to be much more prevalent than gender biases. To spur further research on this topic, we release the sentiment models we used for this study, and the intermediate checkpoints throughout training, yielding 1,525 distinct models; we also release our evaluation code.
Submission Number: 1197
Loading