Evaluating Compositionality in Neural Models Using Arithmetic Expressions

Anonymous

Abstract: We introduce CobA, a dataset designed to evaluate the compositional properties of neural models. The dataset consists of simple arithmetic expressions combining natural integers with addition and multiplication operators. For example, $(5 + 4) \times 2$. We distinguish four aspects of compositionality: localism, substitutivity, productivity, and systematicity. We generate partitions of the dataset with specific in-domain and generalization sets, designed to evaluate the model's ability for each compositional aspect. By carefully selecting expressions from the in-domain and generalization sets, we introduce controlled differences between the two sets. We show that models achieve competitive performance on a random partition, for which there is no controlled difference. Yet, for partitions requiring compositional extrapolation, performances drastically decrease for most encoder architectures. We observe distinctions among architectures, in particular fixed-length context transformers, sequential or tree-structured LSTM.

Paper Type: long