When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoning

Anirban Das, Irtaza Khalid, Rafael Peñaloza, Steven Schockaert

Keywords: Reasoning, Systematic Generalization, Compositional Generalization, Link Prediction, Neuro-symbolic AI

TL;DR: Reasoning models can learn rules from simple examples and be able to solve complex ones using the rules. We identify a broad class of everyday reasoning rules that current models cannot learn and build large datasets requiring such rule learning.

Abstract: Designing models that can learn to reason in a systematic way is an important and long-standing challenge. In recent years, a wide range of solutions have been proposed for the specific case of systematic relational reasoning, including Neuro-Symbolic approaches, variants of the Transformer architecture, and specialized Graph Neural Networks. However, existing benchmarks for systematic relational reasoning focus on an overly simplified setting, based on the assumption that reasoning can be reduced to composing relational paths. In fact, this assumption is hard-baked into the architecture of several recent models, leading to approaches that can perform well on existing benchmarks but are difficult to generalize to other settings. To support further progress in the field of systematic relational reasoning with neural networks, we introduce a new benchmark that adds several levels of difficulty, requiring models to go beyond path-based reasoning.

Dataset URL: https://huggingface.co/datasets/axd353/When-No-Paths-Lead-to-Rome

Code URL: https://github.com/axd353/WhenNoPathsLeadToRome

Primary Area: Other (please use sparingly, only use the keyword field for more details)

Submission Number: 1033