αMax-B-CUBED: A Supervised Metric for Addressing Completeness and Uncertainty in Cluster Evaluation

Felip Guimera Cuevas, Helmut Schmid

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: ClusteringEvaluation metric, Uncertainty, Imprecise and coarse labels, ‘Completeness’ constraint, B-CUBED (B3) precision and recall evaluation metric

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a modified evaluation metric which addresses the issue of 'completeness' constraint in B-CUBED precision and recall evaluation metric when dealing with finer clustering methods or imprecise and coarse labels.

Abstract: Assessing the quality of clustering results is a crucial and challenging task. The B-CUBED ($B^3$) precision and recall evaluation metric has gained popularity due to its ability to meet four formal constraints: homogeneity, completeness, rag bag, and size vs. quantity. However, the 'completeness' constraint, which demands that items of the same category be grouped in the same cluster, can pose problems for finer clustering algorithms that identify sub-clusters within clusters. This issue is particularly pronounced when the available labels are imprecise and coarse, resulting in uncertain and fuzzy cluster evaluations. To address this issue, we propose a modified evaluation metric called $\alpha$Max-$B^3$. Our approach accounts for completeness and uncertainty in subgroup evaluation by reorganizing clusters into super-sets based on the most prevalent label and evaluating them alongside the original clusters using a modified weighted $B^3$ metric. The extent of uncertainty, given by $1-\alpha$, can be either explicitly specified or automatically estimated.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 241