Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Models

ACL ARR 2025 February Submission4164 Authors

Abstract: Language models (LMs) are trained to excel at predicting the next word in the sequence given prior context, and humans also share this predictability in reading comprehension. Neuroscience research reveals that next-word predictability influences brain response, as recorded at millisecond resolution using electroencephalography (EEG). However, little is known about which measures of predictability successfully express the similarity between LMs and humans in the reading comprehension process. Here, we generate regressors for both humans and LMs based on two information measures, including top-1 prediction and surprisal, to predict event-related potential (ERP) elicited from EEG recordings. Our results indicate that while the more advanced LMs show a close correspondence to human performance in word prediction accuracy, only surprisal potentially correlates with language-processing ERPs, especially for open-class words with high semantic content. Moreover, our findings challenge the assumption that scaling LMs with increased parameters and computational budgets will consistently lead to improved convergence with human-like linguistic processing.

Paper Type: Long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics

Research Area Keywords: Cognitive Modeling; Brain Encoding; Reading; Electroencephalography (EEG)

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 4164