MIND SCRAMBLE: UNVEILING LARGE LANGUAGE MODEL PSYCHOLOGY VIA TYPOGLYCEMIA
Keywords: Large Language Model, Typoglycemia, Scrambled Text Understanding
TL;DR: This paper explores whether large language models exhibit human-like cognitive behaviors and mechanisms in derived Typoglycemia scenarios .
Abstract: Although still in its infancy, research into the external behaviors and internal mechanisms of large language models (LLMs) has shown significant promise in addressing complex tasks in the physical world. These studies suggest that powerful LLMs, such as GPT-4, are beginning to exhibit human-like cognitive abilities, including planning, reasoning, and reflection, among others. In this paper, we introduce an innovative research line and methodology named LLM Psychology, which leverages or extends human psychology experiments and theories to investigate cognitive behaviors and mechanisms of LLMs. Practically, we migrate the Typoglycemia phenomenon from psychology to explore the “mind” of LLMs. To comprehend scrambled text in Typoglycemia, human brains rely on context and word patterns, which reveals a fundamental difference from LLMs’ encoding and decoding processes. Through various Typoglycemia experiments at the character, word, and sentence levels, we observe the following: (I) LLMs demonstrate human-like behaviors on a macro scale, such as slightly lower task accuracy with consuming more tokens and time; (II) Different LLMs show varying degrees of robustness to scrambled input, making it a democratized benchmark for model evaluation without crafting new datasets; (III) The impact of different task types varies, with complex logical tasks (e.g., math) in scrambled format being more challenging. Going beyond these, some misleadingly optimistic results suggest that LLMs are still primarily data-driven, and their human-like cognitive abilities may differ from what we perceive; (IV) Interestingly, each LLM exhibit its unique and consistent “cognitive pattern” across various tasks, unveiling a general mechanism in its psychology process. To conclude, we provide an in-depth analysis of hidden layers on a micro scale to explain these phenomena, paving the way for LLMs’ deeper interpretability and future research in LLM Psychology.
Primary Area: interpretability and explainable AI
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3018
Loading