Can Transformers Learn SCFGs? An Analysis of Pretraining on Synthetic Texts

ACL ARR 2025 May Submission3173 Authors

Abstract:

We explore the ability of Transformers to generalize Synchronous Context-Free Grammars (SCFG), i.e. to learn a particular grammar just from example strings.
Two experiments were conducted. The first experiment explored Transformers' capacity to translate between synthetic languages corresponding to the source and target side of an SCFG grammar. The second experiment sought for a Transformer configuration which would be capable of SCFG parsing, i.e. identifying the ability to recognize licensed SCFG pairs of strings based on only positive and negative training examples. With a sufficiently large model, Transformers proved capable to learn this task to a high accuracy (97.8\%) even for very long inputs, longer than any training items.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: nlp, scfg, cfg, parser, parsing, transformer, synthetic data

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: synthetical languages generated by scfgs

Submission Number: 3173