Can Transformers Learn SCFGs? An Analysis of Pretraining on Synthetic Texts

ACL ARR 2026 January Submission10408 Authors

Keywords: transformer, formal, languages, pretraining, translation, parsing, synchronous-context-free-grammars, synthetic-data

Abstract: We explore the ability of Transformers to infer Synchronous Context-Free Grammars (SCFGs), i.e. to learn a particular grammar just from example strings. Two experiments were conducted. The first experiment explored Transformers' capacity to translate between synthetic languages corresponding to the source and target side of an SCFG grammar. The second experiment sought for a Transformer configuration which would be capable of SCFG parsing, i.e. identifying the ability to recognize licensed SCFG pairs of strings based on only positive and negative training examples. With a sufficiently large model, Transformers proved capable to learn this task to a high accuracy (96.70\%) even for very long inputs, longer than any training items. Experiments show limitations and variability that leave parts of the problem open to further research.

Paper Type: Short

Research Area: Mathematical, Symbolic, Neurosymbolic, and Logical Reasoning

Research Area Keywords: formal languages, pretraining, transformers, symbolic, context-free

Languages Studied: synthetic languages

Submission Number: 10408