Humanity in AI: Detecting the Personality of Large Language Models
Abstract: Questionnaires are commonly used to detect the personality of large language models (LLMs). However, LLMs suffer from hallucinations and cannot generate reliable answers making it impossible to detect their true personality through questionnaires. To solve this problem, we propose a new method to detect the personality of LLMs by combining questionnaire and text mining methods in this paper. The text mining method can determine the personality of LLMs based on their generated texts, avoiding the influence of hallucinations. In this paper, we also investigate the source of LLMs' personality by conducting experiments on pre-trained language models (PLMs, such as BERT and GPT) and Chat models (ChatLLMs, such as ChatGPT). The results show that LLMs do contain certain personalities, for example, ChatGPT and ChatGLM exhibit the personality traits of 'Conscientiousness'. Moreover, we find that the personalities of LLMs are derived from their pre-trained data. The instruction data used to train ChatLLMs can enhance the generation of data containing personalities and expose their hidden personality. We compare the results with the human average personality score, and we find that the personality of FLAN-T5 in PLMs and ChatGPT in ChatLLMs is more similar to that of a human, with score differences of 0.34 and 0.22, respectively.
Paper Type: Long
Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics
Research Area Keywords: computational psycholinguistics, cognitive modeling
Contribution Types: Position papers, Theory
Languages Studied: English, Chinese
Submission Number: 463
Loading