From Facts to Folklore: Evaluating Large Language Models on Bengali Cultural Knowledge

ACL ARR 2025 May Submission2960 Authors

Abstract: Recent progress in NLP research has demonstrated remarkable capabilities of large language models (LLMs) across a wide range of tasks. While recent multilingual benchmarks have advanced cultural evaluation for LLMs, critical gaps remain in capturing the nuances of low-resource cultures. Our work addresses these limitations through a Bengali Language Cultural Knowledge (BLanCK) dataset including folk traditions, culinary arts, and regional dialects. Our investigation of several multilingual language models shows that while these models perform well in non-cultural categories, they struggle significantly with cultural knowledge and performance improves substantially across all models when context is provided, emphasizing context-aware architectures and culturally curated training data.

Paper Type: Short

Research Area: Computational Social Science and Cultural Analytics

Research Area Keywords: Computational Social Science and Cultural Analytics, Resources and Evaluation, Ethics, Bias, and Fairness, Question Answering

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Data resources

Languages Studied: Bengali

Submission Number: 2960