Data and Parameter Scaling Laws for Neural Machine TranslationDownload PDF

May 16, 2021 (edited Aug 30, 2021)ACL ARR 2021 May Blind SubmissionReaders: Everyone
  • TL;DR: We observe power law scaling in neural MT and use it to predict BLEU when obtaining more data for low-resource scenarios.
  • Preprint: yes
  • Consent: yes
  • Abstract: We observe that the development cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.
  • Software: zip
0 Replies