Transfer of Structural Knowledge from Synthetic Languages

ACL ARR 2025 February Submission513 Authors

Abstract: This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark — a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: transfer learning / domain adaptation, cross-lingual transfer

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: synthetic languages, English

Submission Number: 513