Generalization Bounds for Magnitude-Based Pruning via Sparse Matrix Sketching

Etash Kumar Guha, Prasanjit Dubey, Xiaoming Huo

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Generalization Bounds, Pruning, Sparsity

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We provide theoretical guarantees on the connection between simple magnitude-based pruning and generalization behavior.

Abstract: Magnitude-based pruning is a popular technique for improving the efficiency of Machine Learning, but also surprisingly maintains strong generalization behavior. Explaining this generalization is difficult, and existing analyses connecting sparsity to generalization rely on more structured compression than simple magnitude-based weight dropping. However, we circumvent the need for structured compression by using recent random matrix theory and sparse matrix sketching results to more tightly tie the connection between pruning-based sparsity and generalization and provide bounds on how Magnitude-Based Pruning and Iterative Magnitude Pruning affects generalization. We empirically verify that our bounds capture the connection between pruning-based sparsity and generalization more than existing bounds.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1144