CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs

ACL ARR 2026 January Submission4724 Authors

Keywords: cultural bias analysis, model bias evaluation, model bias mitigation, ethical considerations in NLP applications, resources for less-resourced languages

Abstract:

As large language models (LLMs) are increasingly deployed in diverse cultural environments, evaluating their cultural understanding capability has become essential for ensuring trustworthy and culturally aligned applications.
However, existing work often suffers from incomplete and insufficiently rich cultural modeling due to the absence of a comprehensive and scientifically grounded approach to cultural modeling,
and many benchmarks remain difficult to scale because they rely heavily on manual data construction.
Building on and synthesizing insights from established cultural theories, and in close collaboration with domain experts, we propose the most comprehensive cultural evaluation framework to date. We construct a novel stratified hierarchical schema comprising 140 dimensions, which provides rigorous guidance for the automated extraction of cultural knowledge and the construction of corresponding evaluation datasets for any cultural context.
Experimental results on constructed datasets demonstrate that our method can effectively evaluate cultural understanding. They also reveal that existing LLMs lack comprehensive cultural competence, which requires dedicated training.
All code and data files are available at
\url{https://github.com/AnonymousUserForSubmission/Culture}

Paper Type: Long

Research Area: Computational Social Science, Cultural Analytics, and NLP for Social Good

Research Area Keywords: cultural bias analysis, NLP tools for social analysis, Sociolinguistics

Contribution Types: Model analysis & interpretability, Approaches low compute settings-efficiency, Data resources, Data analysis

Languages Studied: English, Spanish, Chinese, French, Korean

Submission Number: 4724