On the Effectiveness of Dataset Embeddings in Mono-lingual, Multi-lingual and Zero-shot Conditions

Rob van der Goot, Ahmet Üstün, Barbara Plank

Abstract:

Recent complementary strands of research
have shown that leveraging information on the
data source through encoding their properties
into embeddings can lead to performance increase when training a single model on heterogeneous data sources. However, it remains unclear in which situations these dataset embeddings are most effective, because they are used
in a large variety of settings, languages and
tasks. Furthermore, it is usually assumed that
gold information on the data source is available, and that the test data is from a distribution seen during training. In this work, we
compare the effect of dataset embeddings in
mono-lingual settings, multi-lingual settings,
and with predicted data source label in a zeroshot setting. We evaluate on three morphosyntactic tasks: morphological tagging, lemmatization, and dependency parsing, and use
104 datasets, 66 languages, and two different
dataset grouping strategies. Performance increases are highest when the datasets are of the
same language, and we know from which distribution the test-instance is drawn. In contrast,
for setups where the data is from an unseen distribution, performance increase vanishes