Fast Random Fourier Features Through Randomized Block-Diagonal Projection

TMLR Paper3050 Authors

Abstract: Sparsity has often been used to address expensive matrix multiplication with very large projection matrices for random Fourier features. Traditional sparse projections, however, lead to information loss when sparsity is moderate. In this work, we propose Randomized Block-Diagonal Projection (RBDP) with structured sparsity in a block-diagonal projection matrix and feature shuffling to retain all information of the original features after projection in projected space with high sparsity and efficiency due to the proposed block-diagonal matrix for projection. Error bounds are given in our analysis for the proposed structured sparsity and feature shuffling. We show that our estimators for kernel approximations and random projection are unbiased with the variance inversely proportional to $k$. The proposed method allows much reduced computation with improved complexity $O(\max\{k,D\}n)$, the dimensionality of the feature vector $D$, the dimensionality in the projected space $k$ and the sample size $n$, compared to the complexity $O(kDn)$ where $k\gtrsim 1000$ to keep errors acceptable traditionally associated with random Fourier features and random projection. It is demonstrated in our experiments that the proposed method achieves significant speed improvements, i.e. a speed-up up to 10,000x over Random Kitchen Sinks and a speed-up up to 15x over Fastfood \citep{le13} on real-world datasets. RBDP is a general framework, simple to implement without reliance on the fast Walsh-Hadamard transform, for any shift-invariant kernels with no assumption on the use of Gaussians in the projection matrix. Our code is made available at https://anonymous\footnote{Due to anonymity for the review, the link to the code repository will be provided after the review process.}.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

The paper was desk-rejected due to font issues so font types are corrected in this version:

Desk Rejection Editors In Chief (Kyunghyun Cho, Hugo Larochelle, [e-mail], Gautam Kamath, +1 more) 01 Jun 2024, 09:20 Editors In Chief, Action Editors, Reviewers, Authors
Desk Reject Comments:
Modified template from default, e.g., fonts. Please revisit and resubmit.

Assigned Action Editor: ~Florent_Krzakala1

Submission Number: 3050