TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling

ACL ARR 2025 May Submission4319 Authors

Abstract: The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model-and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation for building high-quality NLP resources in low-resource language/domain pairs.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: language model, machine translation, multi-gpu, synthetic data, corpus

Contribution Types: Publicly available software and/or pre-trained models, Data resources

Languages Studied: French

Submission Number: 4319