Pairwise proximity metrics for topic modelling evaluation based on BERT embeddings.

Anonymous

Abstract: The use of topic modelling methods is a popular way to describe natural language text with a representative set of words. In order to evaluate such methods, objective metrics such as coherence and silhouette scores are commonly used. However, it has been shown that topic assessment based on such metrics does not align well with human judgment for classical document corpora such as articles, books and server logs and, at the same time, it is still unclear how appropriate they are for dialog data. In this paper, we investigate the most commonly used topic modelling evaluation scores in terms of their alignment with human judgment in the specific area of dialog speech. We show that there is still space for improvement in the objective evaluation of topic modelling, and propose a new group of metrics, called Pairwise Proximity metrics, that are shown to align better with human judgment, when compared to coherence and silhouette scores.