Humanity in AI: Assessing LLMs' Personalities through Psychological Features
Abstract: Questionnaire is a common method for detecting the personality of Large Language Models (LLMs). However, their reliability is often compromised by two main issues: hallucinations (LLMs produce inaccurate or irrelevant responses) and the sensitivity of responses to the order of the presented options. To address these issues, we propose combining psychological method with questionnaire. By extracting psychological features from the LLMs' responses, this method can remain unaffected by hallucinations. By normalizing the scores from both methods, this method can obtain an reliability results. We conduct experiments on both pre-trained language models (PLMs), such as BERT and GPT, as well as conversational models (ChatLLMs), such as ChatGPT. The results show that LLMs do contain certain personalities, for example, ChatGPT and ChatGLM exhibit the high score on the traits of 'Conscientiousness'. Additionally, the results also show that the personalities of LLMs are derived from their pre-trained data, human preference alignment can help align the personalities of LLMs more closely with the average traits of human personalities. We compare the results with the human average personality score, and we find that the personality of FLAN-T5 in PLMs and ChatGPT in ChatLLMs is more similar to that of a human, with score differences of 0.34 and 0.22, respectively. We also calculate root mean square error, the results confirm the effectiveness of our method.
Paper Type: Long
Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics
Research Area Keywords: cognitive modeling, computational psycholinguistics
Contribution Types: Data analysis, Position papers, Theory
Languages Studied: English, Chinese
Submission Number: 906
Loading