An Isotropy Analysis in the Multilingual BERT Embedding Space

Anonymous

Abstract: Several studies have explored various advantages of multilingual pre-trained models (such as multilingual BERT) in capturing shared linguistic knowledge. However, less attention has been paid to their limitations. In this paper, we investigate the multilingual BERT for two known issues of the monolingual models: anisotropic embedding space and outliers. We show that, unlike its monolingual counterpart, the multilingual model exhibits no outlier dimension in its representations while it has a highly anisotropic space. Furthermore, our experimental results demonstrate that increasing the isotropy of multilingual space can significantly improve its representation power and performance, similarly to what had been observed for monolingual CWRs. Our analysis indicates that, although the degenerated directions vary in different languages, they encode similar linguistic knowledge, suggesting a shared linguistic space among languages.

Paper Type: short