Using the Omega Index for Evaluating Abstractive Community Detection

Gabriel Murray, Giuseppe Carenini, Raymond T. Ng

Abstract: Numerous NLP tasks rely on clustering or community detection algorithms. For many of these tasks, the solutions are disjoint, and the relevant evaluation metrics assume nonoverlapping clusters. In contrast, the relatively recent task of abstractive community detection (ACD) results in overlapping clusters of sentences. ACD is a sub-task of an abstractive summarization system and represents a twostep process. In the first step, we classify sentence pairs according to whether the sentences should be realized by a common abstractive sentence. This results in an undirected graph with sentences as nodes and predicted abstractive links as edges. The second step is to identify communities within the graph, where each community corresponds to an abstractive sentence to be generated. In this paper, we describe how the Omega Index, a metric for comparing non-disjoint clustering solutions, can be used as a summarization evaluation metric for this task. We use the Omega Index to compare and contrast several community detection algorithms.