We introduced a metric based on the distance between connectivity and consensus matrices to rank clustering algorithms, called the DISCOTEC. Overall, this metric works as intended and tends to select clustering models that are most similar to the consensus. We therefore suggest, as validated through experiments, that a diverse pool of clustering algorithms is required to get the most out of the DISCOTEC. In other words, the more the merrier when going to the disco.

We have shown experimentally that among several choices of distances, the most efficient is to binarise the consensus matrix with respect to its mean and compute its difference with the connectivity matrix. In general, the resulting performance is equal to or better than other ensemble clustering baselines such as the average ARI. The main difference with this baseline is that the DISCOTEC is faster to compute with respect to the number of models. Compared to other internal metrics, the advantage of the DISCOTEC is its tolerance to any type of clustering algorithm, \ie definition of clusters. Consequently, the DISCOTEC shows better performance when the ranking a diverse set of clustering algorithms. In the case of a single clustering algorithm with limited parameters, a specialised internal metric may be preferred.

Finally, we have shown that the DISCOTEC can be regularised with must-link/cannot-link constraints thanks to the approximate measure of informativeness. Moreover, both methods are compatible from a dimensional analysis perspective because they average differences between edges of connectivity matrices.

In future work, it would be interesting to investigate how to further improve the performance of the DISCOTEC when the pool of base clusterings is not diverse. Additionally, it would be interesting to explore different approaches to the raw binarisation of the consensus matrix, \eg a nonlinear bijection to obtain extreme values without being binary.