A New Cluster Validation Index Based on Stability Analysis

Adane Nega Tarekegn, Bjørnar Tessem, Fazle Rabbi

Abstract: Clustering is a frequently employed technique across various domains, including anomaly detection, recommender systems, video analysis, and natural language processing. Despite its broad application, validating clustering results has become one of the main challenges in cluster analysis. This can be due to factors such as the subjective nature of clustering evaluation, lack of ground truth in many real-world datasets, and sensitivity of evaluation metrics to different cluster shapes and algorithms. While there is extensive literature work in this area, developing an evaluation method that is both objective and quantitative is still challenging task requiring more effort. In this study, we proposed a new Clustering Stability Assessment Index (CSAI) that can provide a unified and quantitative approach to measure the quality and consistency of clustering solutions. The proposed CSAI validation index leverages a data resampling approach and prediction analysis to assess clustering stabil