A Study of Cross-lingual Transfer in Continual Slot Filling for Natural Language Understanding

Anonymous

Keywords: continual learning, transfer learning, task oriented dialog systems

Abstract:

Knowledge transfer between neural language models is a widely used technique that has proven to improve performance in a multitude of natural language tasks.
In recent years, high cross-lingual transfer has been shown to occur in multilingual language models.
Hence, it is of great importance to better understand this phenomenon as well as its limits.
While most studies focus on training on independent and identically distributed (i.e. i.i.d.) samples, in this paper we study cross-lingual transfer in continual slot filling for natural language understanding.
We investigate this by training multilingual BERT on one language at a time in sequence from the MultiATIS++ corpus, that contains a total of 9 languages.
Our main findings are that forward transfer is retained although forgetting is still present, and that lost performance can be recovered with as little as a single training epoch.
This may be explained by a progressive shift of model parameters towards a better multilingual initialization.
We also find that commonly used metrics might be insufficient to describe continual learning performance.