Interpreting and Auditing Biases between Bengali Cultural Dialects in Large Language Models with Evaluation and Mitigation Strategies
Abstract: Though Large Language Models (LLMs) have created a massive technological impact, allowing for human-enabled applications, they have the potential to exhibit stereotypes and biases, particularly when dealing with low-resource languages and sensitive topics like cultural differences.
We investigate cultural bias in LLMs by evaluating their performance on Hindu and Muslim-majority cultural dialects of Bengali, and extend this with a user satisfaction study. Through human-centric evaluation and cultural analytics, we assess ChatGPT, Gemini, and Microsoft Copilot using a curated dataset to analyze their handling of culturally-specific words and mitigation of social biases. Our work contributes to human-centric NLP and LLM auditing by exploring reasons for biases observed and strategies for evaluation and mitigation. We aim to promote fairness in LLMs, considering their global impact with over 300 million speakers worldwide.
Paper Type: Long
Research Area: Human-Centered NLP
Research Area Keywords: human-AI interaction, user-centered design, participatory/community-based NLP, values and culture, human-centered evaluation
Contribution Types: Data resources, Data analysis, Position papers, Surveys
Languages Studied: Bangla (Bengali), English
Submission Number: 1313
Loading