The Learnability of Model-Theoretic Interpretation Functions in Artificial Neural Networks

Adrian Brasoveanu, Jakub Dotlacil

Keywords: Systematicity, Compositionality, Out-of-training-sample Generalization, Artificial Neural Networks, Formal Semantics, Model-Theoretic Interpretation, Entity Type Information, Truth Value Type Information

TL;DR: We study the learnability of model-theoretic interpretation functions. Attention architectures have better parameter efficiency than recurrent on small datasets if we use rich, theoretically-motivated semantic representations in training.

Abstract: We investigate the non-symbolic learnability of model-theoretic interpretation functions, revisiting the connectionist systematicity debate in Fodor and Pylyshyn (1988) through the paradigm of Frank et al (2009). We test whether modern architectures (LSTM, Attention/Transformer) outperform simple recurrent networks on out-of-training-sample compositional generalization. Inspired by the fact that entities and truth values are both theoretical primitives in formal semantics, we contrast models trained on purely truth-conditional meaning vectors with models trained on vectors augmented to explicitly include entity information. We find that architectural sophistication leads to better parameter efficiency when leveraging richer, theoretically-motivated semantic representations. The result suggests that progress on generalizable, systematic non-symbolic learning of symbolic semantic notions depends as much on providing theoretically-grounded, richer semantic representation targets as it does on architectural innovation.

Submission Number: 31