Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQBench

TMLR Paper8722 Authors

Abstract:

Cultural Intelligence (CQ) refers to the ability to understand unfamiliar cultural contexts—a crucial skill for large language models (LLMs) to effectively engage with globally diverse users. Existing studies often focus on explicitly stated cultural norms, but fail to capture the subtle, implicit values that are common in daily conversation. To address this gap, we introduce CQ-Bench, a benchmark specifically designed to assess LLMs’ capability to infer implicit cultural values from natural conversational contexts. CQ-Bench consists of multi-character conversation-based stories using values from the World Value Survey and the GlobalOpinions, with topics including ethical, religious, social, etc. Our automatic dataset construction pipeline integrates rigorous validation procedures (incorporation, consistency,
and implicitness checks), achieving a 94.5% human–model agreement in the final validation. To leverage CQ-Bench data, we design three tasks of increasing complexity: attitude detection, value selection, and value extraction. These tasks evaluate whether models can
detect attitude and recognize values embedded within natural dialogues rather than relying on explicit cultural knowledge. We find that while frontier models could reach human-level performance in value selection (0.809 F1), they still fall short in nuanced attitude detection (0.622 F1). Notably, fine-tuning a smaller LLaMA-3.2-3B on only 500 culturally-rich examples improves performance by over 10%, even outperforming o3-mini in some cases. Using CQ-Bench, we provide insights into the current challenges in LLMs’ CQ research
and suggest practical pathways for enhancing LLMs’ cross-cultural reasoning abilities.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Ilia_Sucholutsky1

Submission Number: 8722