A New Rejection Sampling Approach to $k\text{-}\mathtt{means}$++ with Improved Tradeoffs

Poojan Chetan Shah, Shashwat Agrawal, Ragesh Jaiswal

TL;DR: In this work, we present a simple and effective rejection sampling based approach for speeding up k-means++ with improved trade-offs between computational cost and solution quality

Abstract:

The $k$-$\mathtt{means}$++ seeding algorithm (Arthur & Vassilvitskii, 2007a) is widely used in practice for the $k$-means clustering problem where the goal is to cluster a dataset $\mathcal{X} \subset \mathbb{R} ^d$ into $k$ clusters.
The popularity of this algorithm is due to its simplicity and provable guarantee of being $O(\log k)$ competitive with the optimal solution in expectation. However, its running time is $O(|\mathcal{X}|kd)$, making it expensive for large datasets.
In this work, we present a simple and effective rejection sampling based approach for speeding up $k$-$\mathtt{means}$++.
Our first method runs in time $\tilde{O}(\mathtt{nnz} (\mathcal{X}) + \beta k^2d)$
while still being $O(\log k )$ competitive in expectation. Here, $\beta$ is a parameter which is the ratio of the variance of the dataset to the optimal $k$-$\mathtt{means}$ cost in expectation and $\tilde{O}$ hides logarithmic factors in $k$ and $|\mathcal{X}|$.
Our second method presents a new trade-off between computational cost and solution quality. It incurs an additional scale-invariant factor of $ k^{-\Omega( m/\beta)} \operatorname{Var} (\mathcal{X})$ in addition to the $O(\log k)$ guarantee of $k$-$\mathtt{means}$++ improving upon the result of (Bachem et al., 2016a) who get an additional factor of $m^{-1}\operatorname{Var}(\mathcal{X})$ while still running in time $\tilde{O}(\mathtt{nnz}(\mathcal{X}) + mk^2d)$. We perform extensive empirical evaluations to validate our theoretical results and to show the effectiveness of our approach on real datasets.

Primary Area: General Machine Learning->Clustering

Keywords: clustering, k-means++, rejection-sampling

Submission Number: 9401