Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models

James Michaelov, Catherine Arnett, Tyler A. Chang, Ben Bergen

Submission Type: Regular Long Paper

Submission Track: Language Modeling and Analysis of Language Models

Submission Track 2: Linguistic Theories, Cognitive Modeling, and Psycholinguistics

Keywords: abstraction, representation, multilingual language models, psychlinguistics, linguistic structure

TL;DR: Multilingual language models exhibit both within-language and across-language structural priming effects

Abstract: Abstract grammatical knowledge—of parts of speech and grammatical patterns—is key to the capacity for linguistic generalization in humans. But how abstract is grammatical knowledge in large language models? In the human literature, compelling evidence for grammatical abstraction comes from structural priming. A sentence that shares the same grammatical structure as a preceding sentence is processed and produced more readily. Because confounds exist when using stimuli in a single language, evidence of abstraction is even more compelling from crosslingual structural priming, where use of a syntactic structure in one language primes an analogous structure in another language. We measure crosslingual structural priming in large language models, comparing model behavior to human experimental results from eight crosslingual experiments covering six languages, and four monolingual structural priming experiments in three non-English languages. We find evidence for abstract monolingual and crosslingual grammatical representations in the models that function similarly to those found in humans. These results demonstrate that grammatical representations in multilingual language models are not only similar across languages, but they can causally influence text produced in different languages.

Submission Number: 3911