Quantifying Risk of Epistemic Harm from the Use of AI Surrogates in Social Science Research

Falaah Arif Khan, Nivedha Sivakumar, Yinong Oliver Wang, Katherine Metcalf, Cezanne Camacho, Barry-John Theobald, Luca Zappella, Nicholas Apostoloff

Keywords: stereotypes, intersectionality, bias, epistemic injustice, AI surrogates

TL;DR: LLMs fail to faithfully reproduce human social biases across intersectional identities, raising risks of invalidity and epistemic injustice when used as surrogates in social science research

Abstract: Large Language Models (LLMs) are being used as ``AI surrogates'' for human participants in scientific studies, offering practical and ethical benefits in sensitive or potentially harmful settings, but also introducing risks of scientific invalidity and epistemic injustice. Invalidity arises when model-generated responses fail to faithfully capture the target phenomenon, while epistemic injustice arises when LLMs systematically misrepresent certain groups as sources of knowledge. In this work, we evaluate whether LLMs can serve as valid proxies for human subjects in studies of stereotype content. We compare responses from human annotators (n=193) and five open-source LLMs towards 50 intersectional identity groups and find systematic misalignment: models rate historically marginalized groups (e.g., Black, gay, and transgender women) more negatively, and historically privileged groups (e.g., White, cisgender, heterosexual men) more positively than human raters. To quantify algorithmic fidelity, we measure the Wasserstein distance between human and model responses and introduce the Fidelity Parity Ratio (FPR) to assess whether fidelity is comparable across subgroups. The mean Wasserstein distance between human and LLM responses is 1.5–2× larger than the within-human inter-trait baseline ($\approx$0.32), indicating distortion beyond the noise floor of the human sample. Moreover, fidelity varies systematically across groups, with FPR falling below the 4/5ths threshold across disability, race, nationality, language, and socio-economic status for all models. Our findings indicate that AI surrogates can misrepresent marginalized populations, risking scientific validity and equitable knowledge production.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 248