Few-Shot Data-Driven Algorithms for Low Rank Approximation

Piotr Indyk, Tal Wagner, David Woodruff

Keywords: low rank approximation, numerical linear algebra, learning-based algorithms, svd, matrix sketching

TL;DR: We provide new, time and sample efficient, and interpretable algorithms for data-driven low rank matrix approximation. We provide both theoretical results and empirical evaluation for our algorithms.

Abstract:

Recently, data-driven and learning-based algorithms for low rank matrix approximation were shown to outperform classical data-oblivious algorithms by wide margins in terms of accuracy. Those algorithms are based on the optimization of sparse sketching matrices, which lead to large savings in time and memory during testing. However, they require long training times on a large amount of existing data, and rely on access to specialized hardware and software.

In this work, we develop new data-driven low rank approximation algorithms with better computational efficiency in the training phase, alleviating these drawbacks. Furthermore, our methods are interpretable: while previous algorithms choose the sketching matrix either at random or by black-box learning, we show that it can be set (or initialized) to clearly interpretable values extracted from the dataset.

Our experiments show that our algorithms, either by themselves or in combination with previous methods, achieve significant empirical advantage over previous work, improving training times by up to an order of magnitude toward achieving the same target accuracy.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.