The Psychological Effect of Large-Scale Corpus Annotation

ACL ARR 2026 January Submission4528 Authors

Keywords: Annotator well-being, Mental health, Quasi-experimental design, Human factors in NLP

Abstract: Large-scale corpus annotation could have a psychological burden on annotators, and the repeated task of coding text could influence annotators' own attitudes and preferences. Existing work is qualitative, correlational, or inconclusive. Here, we use a quasi-experimental design to investigate the causal relationship between prolonged engagement in data labeling and psychological outcomes. We examined how 30 trained annotators who labeled a large dataset were affected across two important psychological dimensions: mental health and Post Traumatic Stress Disorder (PTSD) symptoms. We found no evidence that annotations led to declines in mental health or PTSD symptoms when compared with a non-annotating control group. Bayesian analyses provided additional support for the null effect. Our findings provide recommendations for future annotation efforts, demonstrating that integrating annotator-centric design principles can safeguard against psychological harm in annotation projects.

Paper Type: Short

Research Area: Computational Social Science, Cultural Analytics, and NLP for Social Good

Research Area Keywords: Computational Social Science and Cultural Analytics, Ethics, Bias, and Fairness, Human-Centered NLP, NLP Applications

Contribution Types: Data analysis

Languages Studied: English

Submission Number: 4528