Sketching Faster than Dimension Times Update Time

William Joseph Swartworth, David Woodruff

Keywords: communication, sketching

Abstract:

In situations such as distributed computation, where one is interested in applying a sketch to a fixed vector, it is often possible to apply the sketch with a runtime that is faster than simulating the corresponding streaming algorithm. For the settings we consider, this avoids an $\omega(1)$ update time lower bound by [Larsen, Nelson, Nguyen '15] when the sketching algorithm has access to the entire vector. We consider a variety of problems.

For the $\ell_2$ heavy-hitters problem, we give a space-optimal sketch which can be applied in linear time (with no logarithmic overhead). We also combine with the ExpanderSketch of [Larsen, Nelson, Nguyen, Thorup'16] to achieve fast decoding time, as well as with a tensor sketch.

For $\ell_p$ estimation with $p\geq 2$ we apply our heavy-hitters scheme to give a linear-time sketch with dimension $\tilde{O}(d^{1-2/p}$), which is nearly optimal.
Using ideas similar to our $\ell_2$-heavy-hitters sketch, we also address linear regression and low-rank approximation, and give sketches that are linear-time in natural regimes.

Finally we introduce a reshaping trick and apply fast matrix multiplication algorithms to speed up $\ell_p$ approxiamtion for $1\leq p \leq 2.$

We discuss applications of these techniques to distributed algorithms.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 19453