Variance Reduction in Sketching Algorithms via Complex Random Variables

Abstract: The seminal \texttt{CountSketch} algorithm of~\cite{count_sketch} compresses high-dimensional real-valued vectors while approximately preserving pairwise inner products in time proportional to the sparsity of the input data. However, the estimator's high variance limits its reliability. In this work, we propose a simple modification of the \texttt{CountSketch} algorithm that not only reduces the variance of the estimate but also maintains its input sparsity running time. Our key idea is to replace real-valued Rademacher $\{-1,1\}$ used in \texttt{CountSketch} with $\{1,\omega,\omega^2,\omega^3\}$(fourth roots of unity). We further extend this idea to the well-known sketching algorithms - \texttt{TensorSketch}~\cite{pham2013fast} and \texttt{Recursive TensorSketch}~\cite{DBLP:conf/soda/AhleKKPVWZ20} for a high-degree polynomial kernel, and obtain improvements in the variance. For \texttt{TensorSketch}, our proposal achieved exponential improvements in the variance, reducing it from $O(3^p)$ to $O(2^p)$. \cite{wacker2022improved} also gives a similar improvement in the variance by exploiting complex random variables. However, the main advantage of our proposal is its running time, which depends on the input sparsity - $ O(p\cdot \mathrm{nnz}(\mathbf{x}))$, whereas for ~\cite{wacker2022improved} it is $O(pd)$, for an input $\bigotimes_{i=1}^{p} \mathbf{x}$ where $\mathbf{x}\in \mathbb{R}^d$. We further extend our technique to \texttt{Recursive TensorSketch}, a state-of-the-art sketching algorithm for polynomial kernels. Our proposal has lower variance than \texttt{Recursive TensorSketch} while retaining the same input sparsity running time.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Lijun_Zhang1

Submission Number: 8571