Components of Creativity: Language Model-based Predictors for Clustering and Switching in Verbal Fluency

Sina Zarrieß, Simeon Junker, Judith Sieker, Özge Alacam

Keywords: verbal fluency; LMs and cognition; creativity; surprisal; attention

TL;DR: We investigate whether recent psychometric predictors computed with transformer language models distinguish between two components of creative semantic search in verbal fluency, namely clustering and switching.

Abstract: Verbal fluency is a well-established experimental paradigm, used to examine various aspects of human knowledge retrieval, linguistic processing, and cognitive performance as well as, more recently, human creative abilities. In this work, we investigate the predictive capacities of recent large language models, known for their ability to store knowledge and retrieve it with high accuracy and efficiency from their latent space. We focus on switching and clustering patterns and seek evidence to substantiate them as two distinct and separable processes in creative semantic search. We prompt different transformer-based language models with verbal fluency items and ask whether metrics derived from the language models' prediction probabilities or internal attention distributions offer reliable predictors of switching/clustering behaviors in verbal fluency. We find that token probabilities, but especially attention-based metrics have strong statistical power when separating between cases of switching and clustering, in line with prior research on human cognition.

Submission Number: 127