A Systematic Evaluation of Transformer-LM Representations for Capturing Author States and Traits

ACL ARR 2024 December Submission1756 Authors

Abstract:

Large Language Models (LLMs) are increasingly used for human-centered tasks, yet their ability to model diverse psychological constructs is not well understood.
This study systematically evaluates the capabilities of diverse Transformer-based LLMs in modeling human psychological constructs across varying levels of temporal stability.
Using a unique dataset of Ecological Momentary Assessments (EMAs) at varying levels of aggregation from none (EMA-level) to waves (quarterly), and users (averaged over ~ 2 years), we explore how autoencoder, encoder-decoder, and autoregressive models capture traits and states.
The findings reveal that the performance of LLMs is influenced by the level of analysis, with models excelling at specific combinations of outcome stability and construct characteristics.
Aggregation strategies play a critical role in enhancing the reliability of predictions for rapidly changing states, moderately stable dispositions, and enduring traits.
These results suggest actionable insights into the design of LLM-based approaches for psychological assessments, emphasizing the importance of selecting appropriate model architectures and temporal aggregation techniques.

Paper Type: Long

Research Area: Human-Centered NLP

Research Area Keywords: Psychological States, Psychological Dispositions, Psychological Traits, Human Behavior, Human-Centered NLP, Computational Social Science

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English

Submission Number: 1756