Personas with Attitudes: Controlling LLMs for Diverse Data Annotation

ACL ARR 2025 February Submission4524 Authors

Abstract: We present a novel approach for enhancing diversity and control in data annotation tasks by personalizing large language models (LLMs). We investigate the impact of injecting diverse persona descriptions into LLM prompts across two studies, exploring whether personas increase annotation diversity and whether the impacts of individual personas on the resulting annotations are consistent and controllable. Our results indicate that persona-prompted LLMs generate more diverse annotations than LLMs prompted without personas, and that the effects of personas on LLM annotations align with subjective differences in human annotations. These effects are both controllable and repeatable, making our approach a valuable tool for enhancing data annotation in subjective NLP tasks such as toxicity detection.

Paper Type: Long

Research Area: Computational Social Science and Cultural Analytics

Research Area Keywords: human behavior analysis; hate-speech detection; psycho-demographic trait prediction; language/cultural bias analysis; NLP tools for social analysis

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 4524