Simulated Replications as a Methodological Tool in Social Science Research

Keywords: Simulated replications, Generative Agent, LLM Simulation

TL;DR: Simulated replications can partly reproduce key social science findings—capturing robust effects but often exaggerating or missing subtle ones—making them a useful but limited complement to traditional methods.

Abstract: Simulation is increasingly recognized as a methodological complement to human-subjects research in the social sciences. This study demonstrates the potential and limitations of simulated data by replicating a published experiment on race cues in mediated communication. Using an AI-enabled workflow, we reproduced the design of Hong et al. [2024], which tested the effects of creator race and influencer race on evaluation, credibility, message acceptance, and engagement. A simulated panel of 240 participants was generated across four experimental conditions and a control group. Statistical analyses showed a partial replication: the main effect of creator race on credibility, a central finding in Hong et al.’s human-sample data, was reproduced and even amplified. Message acceptance again showed null effects. However, the effects of influencer race observed in Hong et al. [2024] were absent, while the effects on evaluation and participation were exaggerated. These results highlight both the promise and the pitfalls of simulation: strong effects may be recoverable, but subtle, context-dependent differences may be lost. More broadly, simulation offers a pathway for accelerating theory testing, replication, and methodological innovation in social science research.

Submission Number: 108