Simulating Social Fingerprints: A DIF-Based Analysis of Structural Bias in LLM Responses

ACL ARR 2025 May Submission7601 Authors

Abstract: Recent studies have explored using large language models (LLMs) as virtual respondents in survey research, with a key challenge being to evaluate how well they align with human responses. This study applies Item Response Theory (IRT) and Differential Item Functioning (DIF)—methods commonly used in human surveys—to analyze item-level bias in LLM-generated responses. IRT estimates a respondent’s latent trait from their answer patterns, while DIF statistically examines whether groups with the same trait respond differently depending on demographic attributes. We constructed personas with various demographic characteristics and simulated their responses to items from the American National Election Studies (ANES). The results show that LLMs replicate human-like bias directions and rankings of influential attributes, but the strength of the bias is substantially amplified. We also observed signs of social desirability bias in LLM responses to race-related items. This study demonstrates that, in the context of persona-assigned LLMs participating in surveys, IRT and DIF analyses enable quantitative, attribute-level bias evaluation—offering a meaningful contribution to the study of human–LLM alignment.

Paper Type: Short

Research Area: Computational Social Science and Cultural Analytics

Research Area Keywords: language/cultural bias analysis, NLP tools for social analysis, model bias/fairness evaluation, LLM/AI agents

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English

Submission Number: 7601