Targeted Syntactic Evaluations with Pseudo-languages

Anonymous

Abstract: In this paper, we propose a novel method called \textbf{Targeted Syntactic Evaluations with pseudo-languages}, which can (i) test whether language models capture linguistic phenomena without relying on other superficial cues (e.g., lexical information), (ii) control the factors out of interest (e.g., vocabulary size), and (iii) easily test linguistic phenomena that only exist in few languages without collecting corpora of those languages. Specifically, we create four types of pseudo-languages with the abstracted vocabulary of different sizes to control the effect of lexical information and vocabulary size, and with different levels of syntactic complexity: $\text{(Adj)}^{n}~\mathrm{NP}$, $\mathrm{NP}^n~\text{VP}^n$, Nested Dependency, and Cross Serial Dependency. We evaluate four different language models (LSTM, BiLSTM, Transformer Encoder, and Transformer Decoder) on these pseudo-languages, using a binary classification of strings based on their grammaticality. Our result demonstrated that the language models have successfully captured the $\text{(Adj)}^{n}~\mathrm{NP}$ type phenomenon irrespective of vocabulary size, while they failed to capture the other phenomena as the vocabulary size increases. These results are not consistent with the previous findings that LSTM or Transformer-based language models can capture syntactic dependencies in natural languages to some extent, suggesting that these language models may not necessarily capture the rules behind these phenomena but rather use some other superficial cues such as co-occurrence or frequency.

Paper Type: long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics