Do Language Differences Lead to Ethical Bias in LLMs? Exploring Dilemmas with the MSQAD and Statistical Hypothesis Tests
Abstract: Despite the recent strides in large language models, studies have underscored the existence of social biases within these systems. In this paper, we delve into the validation and comparison of the multilingual biases of LLM concerning globally discussed and potentially sensitive topics, hypothesizing that these biases may arise from language-specific distinctions. Introducing the Multilingual Sensitive Questions \& Answers Dataset (MSQAD), we compiled news articles from Human Rights Watch covering 17 topics, and generated socially sensitive and controversial questions along with corresponding responses in multiple languages. We scrutinized the biases of these responses across languages and topics, employing various statistical hypothesis tests. The results showed that the null hypotheses were rejected in most cases, indicating a notable cross-language bias. It demonstrates the widespread prevalence of informational bias in responses across diverse languages. By making the proposed MSQAD openly available, we aim to facilitate future research endeavors focused on examining cross-language biases in LLMs and their variant models.
Paper Type: Long
Research Area: Ethics, Bias, and Fairness
Research Area Keywords: Multilingual Bias, LLM, Human Rights
Languages Studied: English, Korean, Chinese, Spanish, German, Hindi
Submission Number: 1592
Loading