CORAL: A Competency-Based Occupational Role-Playing Assessment System for LLMs

ACL ARR 2026 January Submission5334 Authors

Keywords: LLM non-cognitive trait; occupational LLM

Abstract: Current research on Large Language Model (LLM) personalities often overlooks occupational traits and relies on Likert scales, which are prone to bias. To address these issues, this study \textbf{introduces the MAP Occupational Personality Test}, a widely used forced-choice occupational personality scale in China, into the domain of LLM personality assessment. Besides, based on MAP Occupational Personality Test and incorporating open-ended tests, Holland Occupational Themes, this study \textbf{developed the Competency-Based Occupational Role-Playing Assessment System for LLMs (CORAL)}. We evaluated four LLMs (Deepseek-V3, Gemini-2.5-pro, Qwen-3-max, and GPT-4o) across managerial professionals, technological innovators, and high-caliber financial professionals. Results confirm the MAP test’s reliability and validity for LLMs. Notably, the models performed best on open-ended questions, followed by trait understanding, with the poorest performance in occupational motivations. This pattern suggests that LLM proficiency drops from surface-level tasks to deep-level motivations, highlighting the need for future research to focus more on the underlying personality and motivation of these models.

Paper Type: Long

Research Area: Computational Social Science, Cultural Analytics, and NLP for Social Good

Research Area Keywords: LLM occupational non-cognitive ability

Contribution Types: Data analysis

Languages Studied: Chinese; English

Submission Number: 5334