Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs

ACL ARR 2025 February Submission2748 Authors

Abstract: A large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models (LLMs). In this work, we challenge this constrained evaluation paradigm and explore more realistic, unconstrained approaches. Using the World Values Survey (WVS) and Hofstede Cultural Dimensions as case studies, we demonstrate that LLMs exhibit stronger cultural alignment in less constrained settings, where responses are not forced. Additionally, we show that even minor changes, such as reordering survey choices, lead to inconsistent outputs, exposing the limitations of closed-style evaluations. Our findings advocate for more robust and flexible evaluation frameworks that focus on specific cultural proxies, encouraging more nuanced and accurate assessments of cultural alignment in LLMs.

Paper Type: Long

Research Area: Computational Social Science and Cultural Analytics

Research Area Keywords: language/cultural bias analysis, model bias/fairness evaluation, model bias/unfairness mitigation, values and culture, NLP tools for social analysis

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data analysis

Languages Studied: English, German, Bengali

Submission Number: 2748