LLM-Enhanced Survey Methodology: Validation, Automation, and Mixed Methods at Scale

Les DeBusk-Lane, Anirban Pal

Keywords: LLM, Survey Methods

TL;DR: A brief overview of three innovative uses of LLMs in advancing survey research.

Submission Type: Non-Archival

Abstract:

This work explores the transformative potential of large language models (LLMs) across
three key domains in survey research: digital twin simulations, AI-driven telephone
interviewing, and mixed-methods data collection at scale. By leveraging LLMs to
generate synthetic responses mirroring diverse global populations, we assess model
validity in comparison to annual nationally representative survey data from over
140 countries. Trials of LLM-powered phone interviews in the U.S. demonstrate that
AI interviewers can achieve comparable clarity and lower social conformity bias
than human interviewers, while highlighting the need for improved conversational
nuance and participant consent. Finally, we integrate LLMs into mixed-methods
frameworks, using AI-driven conversational probes to deepen qualitative insights
while maintaining methodological rigor. Our findings illustrate the promise of
LLMs in enhancing the efficiency, scalability, and cultural sensitivity of survey
research, alongside ongoing challenges related to bias, methodological replication
and implementation, and technical limitations.

Submission Number: 19