Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations

Philip Blair, Yuval Merhav, Joel Barry

TL;DR: Applying simple heuristics to the Wikidata entity graph results in a high-quality semantic similarity dataset.

Abstract: We propose a language-agnostic way of automatically generating sets of semantically similar clusters of entities along with sets of "outlier" elements, which may then be used to perform an intrinsic evaluation of word embeddings in the outlier detection task. We used our methodology to create a gold-standard dataset, which we call WikiSem500, and evaluated multiple state-of-the-art embeddings. The results show a correlation between performance on this dataset and performance on sentiment analysis.

Keywords: Natural language processing, Applications

Conflicts: basistech.com, neu.edu

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1611.01547/code)