A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing

William Timkey, Tal Linzen

Submission Type: Regular Long Paper

Submission Track: Linguistic Theories, Cognitive Modeling, and Psycholinguistics

Keywords: cue-based retrieval, working memory, interference, attention, agreement attraction, neural networks, cognitive modeling, surprisal, attention

Abstract: Two of the central factors believed to underpin human sentence processing difficulty are expectations and retrieval from working memory. A recent attempt to create a unified cognitive model integrating these two factors have relied on the parallels between the self-attention mechanism of transformer language models and cue-based retrieval theories of working memory in human sentence processing (Ryu and Lewis 2021). While the authors show that attention patterns in specialized attention heads of GPT-2 are consistent with a key prediction of cue-based retrieval models, similarity-based interference effects, their method requires the identification of syntactically specialized attention heads, and makes an cognitively implausible implicit assumption that hundreds of memory retrieval operations take place in parallel. In the present work, we develop a recurrent neural language model with a single self-attention head, which more closely parallels the memory system assumed by cognitive theories. We show that our model’s single attention head can capture semantic and syntactic interference effects observed in human experiments.

Submission Number: 5052