Testing the limits of logical reasoning in neural and hybrid models

Anonymous

TL;DR: We study compositional and recursive generalization in neural networks learning inferences in the syllogistic logic.

Abstract: We study the ability of neural and hybrid models to generalize logical reasoning patterns. We created a series of tests for analyzing various aspects of generalization in the context of language and reasoning, focusing on compositionality and recursiveness. We used them to study the syllogistic logic in hybrid models, where the network assists in premise selection. We analyzed feed-forward, recurrent, convolutional, and transformer architectures. Our experiments demonstrate that even though the models can capture elementary aspects of the meaning of logical terms, they learn to generalize logical reasoning only to a limited degree. Interestingly, the architectures' performance can be unexpected, e.g., convolutional models sometimes generalize better than transformers.

Paper Type: long

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability

Languages Studied: Syllogistic logic