Lessons from using PLMs for Human Cognitive Modeling
Abstract: Many studies show evidence for cognitive abilities in Pre-trained Language Models (PLMs). Researchers have evaluated the cognitive alignment of PLMs, i.e., their correspondence to adult performance across a range of cognitive domains. More recently, the focus has shifted to the developmental alignment of these models: identifying phases during training where improvements in model performance track improvements in children’s thinking over development. However, challenges to this use are twofold: (1) PLMs have very different architectures than human minds and brains, and the data sets on which they are trained differ in many ways from the inputs children receive. (2) The “outputs” of PLMs are different from the behavioral measures that cognitive scientists collect in their experiments and evaluate their theories against. In this paper, we distill lessons learned from using PLMs for cognitive modeling and outline the pitfalls of attempting to use PLMs, not as engineering artifacts, but as cognitive science and developmental science models. We review assumptions used by researchers to map measures of PLM performance to measures of human performances and then, enumerate criteria for using PLMs as credible accounts of cognition and cognitive development.
Paper Type: Short
Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics
Research Area Keywords: cognitive modeling, computational psycholinguistics,
Contribution Types: Position papers
Languages Studied: NA
Submission Number: 3375
Loading