TPR-Attention for Combinatorial Generalization

Melisa Civelekoğlu, Isabeau Prémont-Schwarz

Track: tiny paper (up to 4 pages)

Keywords: TPR, attention, generalization, OOD generalization, combinatorial generalization, systematic generalization, structured representations

TL;DR: We propose a new type of attention mechanism for TPRs which succeeds at combinatorial generalization where normal attention does not.

Abstract: Systematic generalization remains a significant challenge in deep learning. In particular, combinatorial generalization – generalizing to new configurations of known factors of variation – is effortless for humans but difficult for standard neural architectures that rely on statistical correlations rather than explicit structural representations. We introduce a new architectural component that embeds structured inductive bias into deep learning: an attention mechanism operating over tensor‑product representations (TPRs). Through controlled experiments on compositional tasks, we show that this TPR‑attention mechanism outperforms existing architectural components in combinatorial generalization. These results highlight the value of integrating explicit compositional structure into neural attention and point toward a promising path for models capable of systematic generalization.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 77