BaSCo: A Benchmark for Evaluating Social Bias and Cultural Reasoning in Bangla

Afsara Tasnim, Md. Shamsur Rahman, Ajwad Abrar

Keywords: Social Bias, Cultural Reasoning, Large Language Models, Bias Evaluation, Cultural Commonsense, Low-Resource Languages

TL;DR: We introduce BaSCo, a Bangla benchmark showing that reducing bias in LLMs can conflict with culturally grounded reasoning.

Abstract: Large language models (LLMs) are increasingly deployed across diverse linguistic and cultural settings, yet their behavior in low-resource languages such as Bangla remains underexplored. Existing benchmarks primarily focus on high-resource settings and rarely examine the interaction between social bias and culturally grounded reasoning. We introduce BaSCo, a Bangla benchmark for evaluating social bias and cultural reasoning in LLMs, consisting of 2,448 paired instances across age, gender, and hierarchical-relationship categories, where each pair shares the same background context but differs in whether the additional context induces bias or provides culturally grounded information. We evaluate seven LLMs under closed-form and open-ended prompting settings and analyze the impact of prompt-based debiasing. Our results show that closed-form prompting generally reduces bias and improves reliability, whereas open-ended prompting increases biased responses but improves cultural reasoning for stronger models. Prompt-based debiasing reduces bias but increases abstention, revealing a trade-off between fairness and uncertainty. These findings highlight the need for culturally grounded evaluation in Bangla and show that bias mitigation and cultural reasoning should not be treated as independent objectives.

Track: Track 2: ML Research by Muslim Authors

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Non Archival Confirmation: I understand that submissions to MusIML are non-archival and can be submitted to other venues.

Submission Number: 45