Role-Play Enhanced Framework for Big Five Personality Assessment from Counseling Dialogues via Large Language Models

ACL ARR 2024 December Submission18 Authors

Abstract: Accurate assessment of personality traits is crucial for effective psycho-counseling, yet traditional methods like self-report questionnaires are time-consuming and biased. We introduce a novel framework that automatically predicts Big Five (OCEAN) personality traits directly from counseling dialogues by combining role-play prompting with questionnaire-based task decomposition. Our framework conditions Large Language Models (LLMs) to simulate client responses to the Big Five Inventory through counseling dialogue context, achieving significant correlations with professional assessments. Through systematic ablation studies on 853 real-world counseling sessions, we demonstrate that our role-play mechanism significantly improves prediction validity by 33.54\% and reduces safety rejection rates from 28.09\% to 0.31\%. Our fine-tuned LLaMA3-8B model achieves a 36.94\% improvement over larger models like Qwen1.5-110B while reducing computational requirements by 92.73\%. Notably, our framework requires only 30\% of dialogue content for reliable predictions, enabling efficient and unobtrusive personality assessment during natural therapeutic conversations. Our code, models, and data are publicly available to facilitate further research in computational psychometrics.\footnote{\url{https://anonymous.4open.science/r/BigFive-LLM-Predictor-5B41/}}

Paper Type: Long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics

Research Area Keywords: Psycholinguistics, Psychometrics

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data analysis

Languages Studied: Chinese

Submission Number: 18