Quantifying AI Psychology: A Psychometric Benchmark for Large Language Models

ICLR 2025 Conference Submission10533 Authors

Keywords: Large language model, evaluation, psychometrics, psychology

Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities in solving various tasks, progressively evolving into general-purpose assistants. The increasing integration of LLMs into society has sparked interest in whether they exhibit psychological patterns, and whether these patterns remain consistent across different contexts---questions that could deepen the understanding of their behaviors. Inspired by psychometrics, this paper presents a framework for investigating psychology in LLMs, including psychological dimension identification, assessment dataset design, and assessment with results validation. Following this framework, we introduce a comprehensive psychometric benchmark for LLMs that covers five psychological dimensions: personality, values, emotion, theory of mind, and motivation. This benchmark includes 13 datasets featuring diverse scenarios and item types. Our findings suggest that LLMs display a broad spectrum of psychological patterns. We also uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors. This paper offers a thorough psychometric assessment of LLMs, providing insights into reliable evaluation and potential applications in AI and social sciences. Our dataset and code can be accessed via this \href{https://anonymous.4open.science/r/LLM-Psychometrics-Benchmark-2A19}{link}.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10533