Injecting structural hints: Using language models to study inductive biases in language learning

Isabel Papadimitriou, Dan Jurafsky

Submission Type: Regular Long Paper

Submission Track: Linguistic Theories, Cognitive Modeling, and Psycholinguistics

Keywords: transfer learning, pretraining, recursion, context-sensitivity

TL;DR: We use transfer learning from formal languages to natural language in order to understand the effect of the inductive biases of recursion, crossing links, and Zipfian vocabulary on a language learner.

Abstract: Both humans and transformer language models are able to learn language without explicit structural supervision. What cognitive inductive biases make this learning possible? Here, we examine the effect of different inductive learning biases by actively controlling the inductive biases of artificial learners: we structurally bias models by pretraining on synthetic formally-structured data, and evaluate these structural biases by fine-tuning on three typologically-distant human languages: English, Japanese, and Basque. We investigate the effect on downstream language perplexity of three types of inductive bias: 1) recursive, hierarchical processing 2) unrestricted token-token dependencies that can't be modeled by context-free grammars, and 3) a Zipfian power-law vocabulary distribution. We show that complex, non-context-free interactions between tokens form the best inductive biases. Our study leverages the capabilities of transformer models to run controlled language learning experiments that are not possible to run on humans, and surfaces hypotheses about the structures that facilitate language learning in both humans and machines.

Submission Number: 1357