CoreSPECT: Enhancing Clustering Algorithms via an Interplay of Density and Geometry

Chandra Sekhar Mukherjee, Joonyoung Bae, Jiapeng Zhang

TL;DR: We design a clustering enhancement framework based on density-geometry correlations that improves the performance of K-Means and HDBSCAN across many datasets, matching and sometimes exceeding the SOTA algorithms in both accuracy and efficiency.

Abstract:

In this paper, we provide a novel perspective on the underlying structure of real-world data with ground-truth clustering via characterization of an abundantly observed yet often overlooked *density–geometry* correlation.

We leverage this correlation to design CoreSPECT (Core Space Projection based Enhancement of Clustering Techniques), a general framework that improves the performance of generic clustering algorithms. Our framework boosts the performance of clustering algorithms by applying them to strategically selected regions, then extending the partial partition to a complete partition for the dataset using a novel neighborhood graph based multi-layer propagation procedure.

We provide initial theoretical support of the functionality of our framework under the assumption of our model, and then provide large-scale real-world experiments on 20 datasets that include standard image datasets as well as genomics datasets.

We observe two notable improvements. First, CoreSPECT improves the NMI of K-Means by 20 % on average, making it competitive (and in some cases surpassing) the state-of-the art manifold-based clustering algorithms, while being orders of magnitude faster.

Secondly, our framework boosts the NMI of HDBSCAN by more than 100 % on average, making it competitive to the state-of-the-art in several cases *without requiring the true number of clusters and hyper-parameter tuning*. The overall ARI improvements are higher.

Submission Number: 2174