Waste not, Want not: All-Alive Pruning for Extremely Sparse Networks

Daejin Kim, Hyunjung Shim, Jongwuk Lee

Keywords: Model compression, Network pruning, Iterative pruning, Dead connections

Abstract: Network pruning has been widely adopted for reducing computational cost and memory consumption in low-resource devices. Recent studies show that saliency-based pruning can achieve high compression ratios (e.g., 80-90% of the parameters in original networks are removed) without sacrificing much accuracy loss. Nevertheless, finding the well-trainable networks with sparse parameters (e.g., < 10% of the parameters remaining) is still challenging to network pruning, commonly believed to lack model capacity. In this work, we revisit the procedure of existing pruning methods and observe that dead connections, which do not contribute to model capacity, appear regardless of pruning methods. To this end, we propose a novel pruning method, called all-alive pruning (AAP), producing the pruned networks with only trainable weights. Notably, AAP is broadly applicable to various saliency-based pruning methods and model architectures. We demonstrate that AAP equipped with existing pruning methods (i.e., iterative pruning, one-shot pruning, and dynamic pruning) consistently improves the accuracy of original methods at 128× - 4096× compression ratios on three benchmark datasets.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose a simple-yet-effective and versatile unstructured pruning method, namely all-alive pruning (AAP), to eliminate dead connections and make all weights in the subnetwork trainable.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=oXz4jpn0Iw