Humanity in AI: Detecting the Personality of Large Language Models

Anonymous

Abstract: Exploring the personality of large language models (LLMs) is an important way to gain an in-depth understanding of LLMs. It is well known that ChatGPT has reached a level of linguistic proficiency comparable to that of a 9-year-old child, prompting a closer examination of its personality. In this paper, we propose to detect the personality of LLMs by questionnaires and text mining methods, with the guide of BigFive psychological model. To explore the origins of the LLMs personality, we conduct experiments on pre-trained language models (PLMs, such as BERT and GPT) and Chat models (ChatLLMs, such as ChatGPT). The results show that LLMs do contain certain personalities, for example, we think ChatGPT tends to exhibit openness, conscientiousness and neuroticism, while ChatGLM only exhibited conscientiousness and neuroticism. More importantly, we find that the personality of LLMs comes from their pre-training data, and the instruction data can facilitate the generation of data containing personality. We also compare the results of LLMs with the human average personality score, and find that the humanity of FLAN-T5 in PLMs and ChatGPT in ChatLLMs is more similar to that of a human, with score differences of 0.34 and 0.22, respectively.

Paper Type: long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics

Contribution Types: Model analysis & interpretability, Position papers, Theory

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.