TALES: Test Set of Portuguese Lexical-Semantic Relations for AssessingWord Embeddings

Ana Alves

Abstract: Static word embeddings, like word2vec or GloVe, are often assessed when solving syntactic and semantic analogies. Among the latter, we are interested in relations that one would find in lexicalsemantic knowledge bases like WordNet, also covered in analogy test sets for English. This paper describes the creation of a new test for assessing Portuguese word embeddings, dubbed TALES, with an exclusive focus on lexical-semantic relations, acquired from lexical resources in Portuguese. It further reports on the performance of methods previously used for solving analogies, with pre-trained Portuguese word embeddings, when applied to the created dataset, an experiment that revealed that TALES is challenging to solve. Results achieved are briefly discussed, with conclusions that may be useful for developing new approaches for this problem, possibly new embeddings, as well as future versions of TALES.