Reweighted Solutions for Weighted Low Rank Approximation

David Woodruff, Taisuke Yasuda

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Weighted low rank approximation, column subset selection, communication complexity

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Reweighted solutions give a new meaningful class of relaxed solutions to weighted low rank approximation with many benefits

Abstract: The weighted low rank approximation problem is an important yet computationally challenging primitive with applications ranging from statistical analysis, model compression, and signal processing. To cope with the NP-hardness of this problem, prior work either considers heuristics or bicriteria algorithms to solve this problem. In this work, we introduce a new relaxed solution to the weighted low rank approximation which outputs a matrix that is not necessarily low rank, but can be stored using very few parameters and gives provable approximation guarantees for this problem when the rank matrix has low rank. Our central idea is to use the weight matrix itself to reweight the low rank solution. Our algorithm is extremely simple to implement and achieves remarkable empirical performance in applications to model compression. Our algorithm also gives nearly optimal communication complexity bounds for a natural distributed algorithm associated with the low rank approximation problem, for which we show matching communication lower bounds. Together, our communication complexity bounds show that the rank of the weight matrix provably parameterizes the communication complexity of weighted low rank approximation. We also obtain the first feature selection guarantees for weighted low rank approximation.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7701