Contextualized Embeddings in Named-Entity Recognition: An Empirical Study on Generalization

B. Taille, V. Guigue, P. Gallinari

Abstract:

Contextualizedembeddingsuseunsupervisedlanguagemodelpretrain- ing to compute word representations depending on their context. This is intuitively useful for generalization, especially in Named-Entity Recognition where it is crucial to detect mentions never seen during training. However, standard English benchmarks overestimate the importance of lexical over contextual features be- cause of an unrealistic lexical overlap between train and test mentions. In this paper, we perform an empirical analysis of the generalization capabilities of state- of-the-art contextualized embeddings by separating mentions by novelty and with out-of-domain evaluation. We show that they are particularly beneficial for unseen mentions detection, especially out-of-domain. For models trained on CoNLL03, language model contextualization leads to a +1.2% maximal relative micro-F1
1 score increase in-domain against +13% out-of-domain on the WNUT dataset