A Rademacher-Like Random Embedding with Linear Complexity
Keywords: Random Embedding, Randomized Singular Value Deomposition, Randomized Arnoldi Process, Machine Learning, Linear Complexity
Abstract: Random embedding assumes an important role in representation learning. Gaussian embedding and Rademacher embedding are two widely used random embeddings. Although they usually enjoy robustness and effectiveness, their computational complexity is high, i.e. $O(nk)$ for embedding an $n$-dimensional vector into $k$-dimensional space. The alternatives include partial subsampled randomized Hadamard (P-SRHT) embedding and sparse sign embedding, which are still not of linear complexity or cannot run efficiently in practical implementation. In this paper, a fast and robust Rademacher-like embedding (RLE) is proposed, based on a smaller Rademacher matrix and several auxiliary random arrays. Specifically, it embeds an $n$-dimensional vector into $k$-dimensional space in just $O(n)$ time and space (assuming $k$ is not larger than $O(n^{\frac{1}{2}})$). Our theoretic analysis reveals that the proposed RLE owns most of desirable properties of the Rademacher embedding while preserving lower complexity. To validate the practical efficiency and effectiveness, the proposed RLE is applied to single-pass randomized singular value decomposition (single-pass RSVD) for streaming data, and the randomized Arnoldi process based on sketched ordinary least-squares. Numerical experiments show that, with the proposed RLE the single-pass RSVD achieves 1.7x speed-up on average while keeping same or better accuracy, and the randomized Arnodli process enables a randomized GMRES algorithm running 1.3x faster on average for solving $Ax=b$ than that based on other embeddings.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9927
Loading