Fifteen Percent Fluency: Measuring the Cultural Knowledge-Behaviour Gap in LLMs

ACL ARR 2026 January Submission5510 Authors

Keywords: Pragmatic competence, Cultural adaptation and sensitivity, Evaluation methodology, Multilingual analysis, Contextual inference

Abstract: Large language models possess cultural knowledge but deploy it selectively: when given explicit instruction (``respond as someone from a high power-distance culture''), they adapt readily; when the same cultural context is embedded through implicit situational cues, they do not. We introduce a triad evaluation methodology to quantify this gap. For 60 scenarios across three Hofstede dimensions, we collect model responses under neutral (Prompt A), explicit (Prompt B), and implicit (Prompt C) conditions. The ratio of implicit adaptation to explicit capability, Pragmatic Context Sensitivity (PCS), measures what fraction of demonstrated competence models actually use. Across four models spanning frontier and budget tiers and five languages (English, German, Hindi, Nepali, Urdu), mean PCS is 0.15: models deploy only 15\% of their cultural capability when relying on contextual cues alone. This gap is consistent across architectures and dimension-asymmetric: power distance cues elicit 29\% of explicit capability while individualism-collectivism (12\%) and uncertainty avoidance (4\%) show minimal adaptation. A Hindi-Urdu comparison reveals no statistically significant pragmatic divergence ($p = 0.26$, $d = 0.03$), suggesting models respond primarily to linguistic structure rather than cultural indexicality. These findings indicate that current alignment paradigms instil culturally specific defaults that explicit instruction can override but implicit context cannot. Users who most need culturally appropriate communication are precisely those least equipped to request it.

Paper Type: Long

Research Area: Discourse, Pragmatics, and Reasoning

Research Area Keywords: pragmatics, discourse, contextual inference, language use, cross-cultural communication, multilingual discourse

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English, German, Urdu, Hindi, Nepali

Submission Number: 5510