Comparing Fifty Natural Languages and Twelve Genetic Languages Using Word Embedding Language Divergence (WELD) as a Quantitative Measure of Language Distance

Ehsaneddin Asgari, Mohammad Mofrad

Abstract:

We introduce a new measure of distance between languages based on word embedding,
called word embedding language divergence
(WELD). WELD is defined as divergence between unified similarity distribution of words
between languages. Using such a measure,
we perform language comparison for fifty natural languages and twelve genetic languages.
Our natural language dataset is a collection of
sentence-aligned parallel corpora from bible
translations for fifty languages spanning a variety of language families. Although we use
parallel corpora, which guarantees having the
same content in all languages, interestingly in
many cases languages within the same family cluster together. In addition to natural
languages, we perform language comparison
for the coding regions in the genomes of 12
different organisms (4 plants, 6 animals, and
two human subjects). Our result confirms
a significant high-level difference in the genetic language model of humans/animals versus plants. The proposed method is a step toward defining a quantitative measure of similarity between languages, with applications in
languages classification, genre identification,
dialect identification, and evaluation of translations.