Subword Contextual Embeddings for Languages with Rich Morphology

Arda Akdemir, Tetsuo Shibuya, Tunga Güngör

Abstract: Morphological information is important for many sequence labeling tasks in Natural Language Processing (NLP). Yet, existing approaches rely heavily on manual annotations or external software to capture this information. In this study, we propose using subword contextual embeddings for languages with rich morphology. Evaluated on Dependency Parsing (DEP) and Named Entity Recognition (NER) tasks, which are shown to benefit highly from morphological information, subword contextual embeddings consistently outperformed other approaches on all languages tested (Hungarian, Finnish, Czech and Turkish). Our proposed method enables achieving state-of-the-art results with little annotation requirements compared to the previous work. Besides, the novel network architecture we propose, coupled with a Bayesian hyperparameter optimization suite, achieved state-of-the-art results for both tasks for the Turkish language. Finally, we experimented with different multi-task learning architectures to analyze the effect of jointly learning the two tasks.