What they do when in doubt: a study of inductive biases in seq2seq learners

Eugene Kharitonov, Rahma Chaabouni

Keywords: inductive biases, description length, sequence-to-sequence models

Abstract: Sequence-to-sequence (seq2seq) learners are widely used, but we still have only limited knowledge about what inductive biases shape the way they generalize. We address that by investigating how popular seq2seq learners generalize in tasks that have high ambiguity in the training data. We use four new tasks to study learners' preferences for memorization, arithmetic, hierarchical, and compositional reasoning. Further, we connect to Solomonoff's theory of induction and propose to use description length as a principled and sensitive measure of inductive biases. In our experimental study, we find that LSTM-based learners can learn to perform counting, addition, and multiplication by a constant from a single training example. Furthermore, Transformer and LSTM-based learners show a bias toward the hierarchical induction over the linear one, while CNN-based learners prefer the opposite. The latter also show a bias toward a compositional generalization over memorization. Finally, across all our experiments, description length proved to be a sensitive measure of inductive biases.

One-sentence Summary: Standard seq2seq models can infer perfectly different rules when presented with as little as one training example, showing strikingly different inductive biases that can be studied via description length

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Data: [SCAN](https://paperswithcode.com/dataset/scan)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2006.14953/code)