Parsing the Old Using the New and Vice Versa: Fine-tuning Pre-Trained Language Models for Icelandic

Mathias Hans Erik Stenlund, Beáta Megyesi

Keywords: parsing, icelandic, ConvBERT, historical, modern, fine-tune, pre-trained, LAS

TL;DR: A pre-trained model is fine-tuned and evaluated on parsing historical and modern Icelandic.

Abstract: In this study, we present experiments on parsing historical Icelandic by using a pre-trained ConvBERT language model for modern Icelandic which is then fine-tuned on modern Icelandic, historical Icelandic, and on a combination of both. Using the dependency parser DiaParser, the models are evaluated on both modern and historical Icelandic. The results indicate that fine-tuning on in-domain data is ideal; fine-tuning on historical texts when parsing historical texts achieves 82.9% LAS, and fine-tuning and testing on modern text reaches 87.96% LAS. The best performing model is obtained on fine-tuning on a merged dataset, achieving 85% LAS on historical data, and 89% on modern data.

Student Paper: Yes, the first author is a student