Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammars

ACL ARR 2026 May Submission16350 Authors

Keywords: Transformer Grammars, Syntactic Language Models, Tree Traversal, Linearization, Structural Inductive Biases, Language Modeling

Abstract: Transformer Grammars (TGs) enhance language modeling by incorporating syntactic tree structures. Despite the potentially significant impact on model performance of how syntactic trees are linearized in TGs, existing studies rely solely on Depth-First Traversal (DFT) for linearization. In this paper, we expand the traversal design space by exploring Breadth-First Traversal (BFT) and a novel hybrid traversal strategy, Production-Rule Traversal (PRT), which combines the structural lookahead of BFT with the early lexical generation of DFT. We integrate these traversal methods with varying tree configurations and masking strategies, and empirically evaluate their performance on language modeling, syntactic generalization and summarization. We reveal the inherent trade-offs between nested composition and global lookahead, providing actionable recommendations for designing task-aware Transformer Grammars.

Paper Type: Short

Research Area: Syntax: Tagging, Chunking and Parsing

Research Area Keywords: constituency parsing,hierarchical structure prediction,grammar and knowledge-based approaches

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

EMNLP 2026 AI Reviewing Experiment: yes

Submission Number: 16350