Limited Differences in LLM Performance for Social Majorities and Minorities in Survey Synthetic Data

ACL ARR 2026 January Submission7323 Authors

Keywords: computational social science, large language models, synthetic survey data, bias and fairness, majority–minority groups

Abstract: Large Language Models (LLMs) are increasingly used to generate synthetic survey data, yet it remains unclear whether they perform equally well for social minorities and majorities. Using a unique survey conducted in South Korea in 2012 that administered identical questionnaires to adolescents from multicultural and monocultural families, we examine group-level differences in the accuracy of LLM-generated synthetic data. We construct personas that mirror individual-level demographic profiles and prompt two LLMs, llama3:latest and ChatGPT-4o, to answer 77 survey questions across 11 question domains. We find LLMs' limited performance differences between minority and majority groups. Instead, accuracy varies by question domain and LLM. Across all domains, synthetic data generally exhibit lower variance than human responses. These findings highlight both the potential and the limitations of using LLMs to generate synthetic survey data for studying social minorities.

Paper Type: Short

Research Area: Computational Social Science, Cultural Analytics, and NLP for Social Good

Research Area Keywords: psycho-demographic trait prediction, language/cultural bias analysis

Contribution Types: Reproduction study, Data resources, Data analysis, Surveys

Languages Studied: Korean

Submission Number: 7323