People are not their Countries: Alignment Evaluation Beyond Nationality

Maria-Louisa Wightman, Guillaume Bied, Tijl De Bie

Track: Main Papers Track (6 to 9 pages)

Keywords: alignment, language models, evaluation

TL;DR: Evaluating LLM alignment to different socio-demographic groups and thereby extending previous work focusing on nationality.

Abstract: Large Language Models (LLMs) are rapidly developing into a primary source of information. As information is rarely value-neutral, the study of which values are inscribed in LLMs, and the development of methodologies to answer this question, is an increasingly active area of research. Of particular interest is the extent to which the behavior of various LLMs is aligned with the values of different defined populations. Recent research has attempted to study this by quantifying the values of different populations through the World Value Survey, and comparing their responses to those of different LLMs. So far, with a few limited exceptions, these populations have been defined by nationality. In the present paper, we acknowledge that individuals with the same nationality can have wildly differing socio-demographic backgrounds. This may lead to considerable value diversity that remains invisible when lumping these groups together at the country level. As a result, national‑level alignment scores may reflect demographic composition rather than cross‑country differences, masking unequal representation of certain groups by popular LLMs. Relying on the European Social Survey, we address this knowledge gap by studying value alignment at the level of socio-demographic groups across Europe, with demography defined by gender, education, income, age, religion, and more. Our analyses reveal that LLMs are indeed unequally aligned to the values of different socio-demographic groups, with religious denomination standing out as particularly influential.

Submission Number: 60