Optimal pruning of hierarchical clustering dendrograms

Jiacheng Ge, Robert Tibshirani

Abstract: Hierarchical clustering is a popular method for identifying distinct groups in a dataset. The most commonly used method for pruning a dendrogram is via a single horizontal cut. In this article, we propose an “optimal pruning” method. We prove its superiority over horizontal pruning and provide some examples illustrating how the two methods can behave quite differently. Additionally, we compare our approach to dynamic programming. Furthermore, we discuss the selection of the optimal number of clusters.