I. Cross-lingual language model pretraining (XLM)

XLM supports multi-GPU and multi-node training, and contains code for:

• Language model pretraining:
  • Causal Language Model (CLM)
  • Masked Language Model (MLM)
  • Translation Language Model (TLM)
• GLUE fine-tuning
• XNLI fine-tuning
• Supervised / Unsupervised MT training:
  • Denoising auto-encoder
  • Parallel data training
  • Online back-translation

Dependencies

• Python 3
• NumPy
• PyTorch (currently tested on version 0.4 and 1.0)
• fastBPE (generate and apply BPE codes)
• Moses (scripts to clean and tokenize text only - no installation required)
• Apex (for fp16 training)

II. Train your own model

Open the illustrative notebook in colab

Note : Most of the bash scripts used in this repository were written on the windows operating system, and can generate this error on linux platforms.
This problem can be corrected with the following command:

filename=my_file.sh 
cat $filename | tr -d '\r' > $filename.new && rm $filename && mv $filename.new $filename

1. Preparing the data

At this level, if you have pre-processed binary data in pth format (for example from XLM experimentation or improvised by yourself), group them in a specific folder that you will mention as a parameter by calling the script train.py.
If this is not the case, we assume that you have txt files available for preprocessing. Look at the following example for which we have three translation tasks: English-French, German-English and German-French.

We have the following files available for preprocessing:

- en-fr.en.txt and en-fr.fr.txt 
- de-en.de.txt and de-en.en.txt 
- de-fr.de.txt and de-fr.fr.txt

All these files must be in the same folder (PARA_PATH).
You can also (only or optionally) have monolingual data available (en.txt, de.txt and fr.txt; in MONO_PATH folder).
Parallel and monolingual data can all be in the same folder.

Note : Languages must be submitted in alphabetical order (de-en and not en-de, fr-ru and not ru-fr...). If you submit them in any order you will have problems loading data during training, because when you run the train.py script the parameters like the language pair are put back in alphabetical order before being processed. Don't worry about this alphabetical order restriction, XLM for MT is naturally trained to translate sentences in both directions. See translate.py.

OPUS collections is a good source of dataset. We illustrate in the opus.sh script how to download the data from opus and convert it to a text file.
Changing parameters ($PARA_PATH and $SRC) in opus.sh.

cd meta_XLM
chmod +x ./scripts/opus.sh
./scripts/opus.sh de-fr

Another source for other_languages-english data is anki Tab-delimited Bilingual Sentence Pairs. Simply download the .zip file, unzip to extract the other_language.txt file. This file usually contains data in the form of sentence_en sentence_other_language other_information on each line. See anki.py and anky.sh in relation to a how to extract data from anki. Example of how to download and extract de-en and en-fr pair data.

cd meta_XLM
output_path=/content/data/para
mkdir $output_path
chmod +x ./scripts/anki.sh
./scripts/anki.sh de,en deu-eng $output_path scripts/anki.py
./scripts/anki.sh en,fr fra-eng $output_path scripts/anki.py

After that you will have in data/para following files : de-en.de.txt, de-en.en.txt, deu.txt, deu-eng.zip and _about.txt

Move to the XLM folder in advance.

cd XLM

Install the following dependencies (fastBPE and Moses) if you have not already done so.

git clone https://github.com/moses-smt/mosesdecoder tools/mosesdecoder
git clone https://github.com/glample/fastBPE tools/fastBPE && cd tools/fastBPE && g++ -std=c++11 -pthread -O3 fastBPE/main.cc -IfastBPE -o fast

Changing parameters in data.sh. Between lines 94 and 100 of data.sh, you have two options corresponding to two scripts to execute according to the distribution of the folders containing your data. Option 2 is chosen by default, kindly uncomment the lines corresponding to your option.
With too many BPE codes (depending on the size of the dataset) you may get this error. Decrease the number of codes (e.g. you can dichotomously search for the appropriate/maximum number of codes that make the error disappear)

languages=de,en,fr
chmod +x ../data.sh 
../data.sh $languages

If you stop the execution when processing is being done on a file please delete this erroneous file before continuing or restarting the processing, otherwise the processing will continue with this erroneous file and potential errors will certainly occur.

After this you will have the following (necessary) files in $OUTPATH (and $OUTPATH/fine_tune depending on the parameter $sub_task):

- monolingual data :
    - training data   : train.fr.pth, train.en.pth and train.de.pth
    - test data       : test.fr.pth, test.en.pth and test.de.pth
    - validation data : valid.fr.pth, valid.en.pth and valid.de.pth
- parallel data :
    - training data : 
        - train.en-fr.en.pth and train.en-fr.fr.pth 
        - train.de-en.en.pth and train.de-en.de.pth
        - train.de-fr.de.pth and train.de-fr.fr.pth 
    - test data :
        - test.en-fr.en.pth and test.en-fr.fr.pth 
        - test.de-en.en.pth and test.de-en.de.pth
        - test.de-fr.de.pth and test.de-fr.fr.pth 
    - validation data
        - valid.en-fr.en.pth and valid.en-fr.fr.pth 
        - valid.de-en.en.pth and valid.de-en.de.pth
        - valid.de-fr.de.pth and valid.de-fr.fr.pth 
 - code and vocab

To use the biblical corpus, run bible.sh instead of data.sh. Here is the list of languages available (and to be specified as $languages value) in this case :

• Languages with data in the New and Old Testament : Francais, Anglais, Fulfulde_Adamaoua or Fulfulde_DC (formal name : Fulfulde), Bulu, KALATA_KO_SC_Gbaya or KALATA_KO_DC_Gbaya (formal name : Gbaya), BIBALDA_TA_PELDETTA (formal name : MASSANA), Guiziga, Kapsiki_DC (formal name : Kapsiki), Tupurri.
• Languages with data in the New Testament only : Bafia, Ejagham, Ghomala, MKPAMAN_AMVOE_Ewondo (formal name : Ewondo), Ngiemboon, Dii, Vute, Limbum, Mofa, Mofu_Gudur, Doyayo, Guidar, Peere_Nt&Psalms, Samba_Leko, Du_na_sdik_na_wiini_Alaw.
  It is specified in bible.sh that you must have in csv_path a folder named csvs. Here is the drive link of its zipped version.
  Concerning training, specify the first four letters of each language (Bafi instead of Bafia for example), except KALATA_KO_SC_Gbaya/KALATA_KO_DC_Gbaya which becomes Gbay (first letters of Gbaya), BIBALDA_TA_PELDETTA which becomes MASS (first letters of MASSANA), MKPAMAN_AMVOE_Ewondo which becomes Ewon (first letters of Ewondo), Francais and Anglais which becomes repectively fr and en. Indeed, bible.sh uses these abbreviations to create the files and not the language names themselves.
  One last thing in the case of the biblical corpus is that when only one language is to be specified, it must be specified twice. For example: languages=Bafia,Bafia instead of languages=Bafia.

2. Pretrain a language model

Install the following dependencie (Apex) if you have not already done so.

git clone https://github.com/NVIDIA/apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./apex

Instead of passing all the parameters of train.py, put them in a json file and specify the path to this file in parameter (See lm_template.json file for more details).

config_file=../configs/lm_template.json
python train.py --config_file $config_file

If you pass a parameter by calling the script train.py (example: python train.py --config_file $config_file --data_path my/data_path), it will overwrite the one passed in $config_file.
Once the training is finished you will see a file named train.log in the $dump_path/$exp_name/$exp_id folder information about the training. You will find in this same folder your checkpoints and best model.
When "mlm_steps":"...", train.py automatically uses the languages to have "mlm_steps":"de,en,fr,de-en,de-fe,en-fr" (give a precise value to mlm_steps if you don't want to do all MLM and TLM, example : "mlm_steps":"en,fr,en-fr"). This also applies to "clm_steps":"..." which deviates "clm_steps":"de,en,fr" in this case.

Note :
-en means MLM on en, and requires the following three files in data_path: a.en.pth, a ∈ {train, test, valid} (monolingual data)
-en-fr means TLM on en and fr, and requires the following six files in data_path: a.en-fr.b.pth, a ∈ {train, test, valid} and b ∈ {en, fr} (parallel data)
-en,fr,en-fr means MLM+TLM on en, fr, en and fr, and requires the following twelve files in data_path: a.b.pth and a.en-fr.b.pth, a ∈ {train, test, valid} and b ∈ {en, fr}

To train with multiple GPUs use:

export NGPU=8; python -m torch.distributed.launch --nproc_per_node=$NGPU train.py --config_file $config_file

Tips: Even when the validation perplexity plateaus, keep training your model. The larger the batch size the better (so using multiple GPUs will improve performance). Tuning the learning rate (e.g. [0.0001, 0.0002]) should help.

Description of some essential parameters

## main parameters
exp_name                     # experiment name
exp_id                       # Experiment ID
dump_path                    # where to store the experiment (the model will be stored in $dump_path/$exp_name/$exp_id)

## data location / training objective
data_path                    # data location 
lgs                          # considered languages/meta-tasks
clm_steps                    # CLM objective
mlm_steps                    # MLM objective

## transformer parameters
emb_dim                      # embeddings / model dimension
n_layers                     # number of layers
n_heads                      # number of heads
dropout                      # dropout
attention_dropout            # attention dropout
gelu_activation              # GELU instead of ReLU

## optimization
batch_size                   # sequences per batch
bptt                         # sequences length
optimizer                    # optimizer
epoch_size                   # number of sentences per epoch
max_epoch                    # Maximum epoch size
validation_metrics           # validation metric (when to save the best model)
stopping_criterion           # end experiment if stopping criterion does not improve

## dataset
#### These three parameters will always be rounded to an integer number of batches, so don't be surprised if you see different values than the ones provided.
train_n_samples              # Just consider train_n_sample train data
valid_n_samples              # Just consider valid_n_sample validation data 
test_n_samples               # Just consider test_n_sample test data for
#### If you don't have enough RAM/GPU or swap memory, leave these three parameters to True, otherwise you may get an error like this when evaluating :
###### RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered
remove_long_sentences_train # remove long sentences in train dataset      
remove_long_sentences_valid # remove long sentences in valid dataset  
remove_long_sentences_test  # remove long sentences in test dataset

There are other parameters that are not specified here (see train.py)

3. Train a (unsupervised/supervised) MT from a pretrained model

See mt_template.json file for more details.

config_file=../configs/mt_template.json
python train.py --config_file $config_file

When the ae_steps and bt_steps objects alone are specified, this is unsupervised machine translation, and only requires monolingual data. If the parallel data is available, give mt_step a value based on the language pairs for which the data is available.

Description of some essential parameters

The description made above remains valid here

## main parameters
reload_model     # model to reload for encoder,decoder
## data location / training objective
ae_steps          # denoising auto-encoder training steps
bt_steps          # back-translation steps
mt_steps          # parallel training steps
word_shuffle      # noise for auto-encoding loss
word_dropout      # noise for auto-encoding loss
word_blank        # noise for auto-encoding loss
lambda_ae         # scheduling on the auto-encoding coefficient

## transformer parameters
encoder_only      # use a decoder for MT

## optimization
tokens_per_batch  # use batches with a fixed number of words
eval_bleu         # also evaluate the BLEU score

There are other parameters that are not specified here (see train.py)

4. How to evaluate a language model trained on a language L on another language L'.

Our

Prerequisite

If you want to evaluate the LM on a language lang, you must first have a file named lang.txt in the $src_path directory of eval_data.sh.
For example if you want to use the biblical corpus, you can run scripts/bible.py :

# folder containing the csvs folder
csv_path=
# folder in which the objective folders will be created (mono or para)
output_dir=
# monolingual one ("mono") or parallel one ("para")
data_type=mono
# list of languages to be considered in alphabetical order and separated by a comma
# case of one language
languages=lang,lang  
# case of many languages
languages=lang1,lang2,...   
# old_only : use only old testament
#  use only new testament
new_only=True

python ../scripts/bible.py --csv_path $csv_path --output_dir $output_dir --data_type $data_type --languages $languages --new_only $new_only

See other parameters in scripts/bible.py

Data pre-processing

Modify parameters in eval_data.sh

# languages to be evaluated
languages=lang1,lang2,... 
chmod +x ../eval_data.sh 
../eval_data.sh $languages

Evaluation

We take the language to evaluate (say Bulu), replace the files test.Bulu.pth (which was created with the VOCAB and CODE of Bafi, the evaluating language) with test.Bafi.pth (since Bafi evaluates, the train.py script requires that the dataset has the (part of the) name of the lgs). Then we just run the evaluation, the results (acc and ppl) we get is the result of LM Bafia on the Bulu language.

# evaluating language
tgt_pair=
# folder containing the data to be evaluated (must match $tgt_path in eval_data.sh)
src_path=
# You have to change two parameters in the configuration file used to train the LM which evaluates ("data_path":"$src_path" and "eval_only": "True")
# You must also specify the "reload_model" parameter, otherwise the last checkpoint found will be loaded for evaluation.
config_file=../configs/lm_template.json 
# languages to be evaluated
eval_lang= 
chmod +x ../scripts/evaluate.sh
../scripts/evaluate.sh $eval_lang

When the evaluation is finished you will see a file named eval.log in the $dump_path/$exp_name/$exp_id folder containing the evaluation results.
Note :The description given below is only valid when the LM evaluator has been trained on only one language (and therefore without TLM). But let's consider the case where the basic LM has been trained on en-fr and we want to evaluate it on de or de-ru. $tgt_pair deviates from en-fr, but language varies depending on whether the evaluation is going to be done on one language or two:

• In the case of de : lang=de-de
• in the case of de-ru: lang=de-ru.

IV. References

Please cite [1] and [2] if you found the resources in this repository useful.

Cross-lingual Language Model Pretraining

[1] G. Lample , A. Conneau Cross-lingual Language Model Pretraining and facebookresearch/XLM

* Equal contribution. Order has been determined with a coin flip.

@article{lample2019cross,
  title={Cross-lingual Language Model Pretraining},
  author={Lample, Guillaume and Conneau, Alexis},
  journal={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2019}
}

License

See the LICENSE file for more details.