Towards Dynamic Sparsification by Iterative Prune-Grow LookAheads

Xinglong Sun, Maying Shen, Hongxu Yin, Lei Mao, Pavlo Molchanov, Jose M. Alvarez

Keywords: Network pruning, Network growing, Efficient Networks

TL;DR: Network pruning via an iterative Grow-and-Prune approach

Abstract: Model sparsification is a process of removing redundant connections in a neural network, making it more compact and faster. Most pruning methods start with a dense pretrained model, which is computationally intensive to train. Other pruning approaches perform compression at initialization which saves training time, however, at the cost of final accuracy as an unreliable architecture can be selected given weak feature representation. In this work, we re-formulate network sparsification as an exploitation-exploration process during initial training to enable dynamic learning of network sparsification. The exploitation phase assumes architecture stability and trains it to maximize accuracy. Whereas the exploration phase challenges the current architecture with a novel $\textit{LookAhead}$ step that reactivates pruned parameters, quickly updates them together with existing ones, and reconfigures the sparse architecture with a pruning-growing paradigm. We demonstrate that $\textit{LookAhead}$ methodology can effectively and efficiently oversee both architecture and performance during training, enabling early pruning with a capability of future recovery to correct previous poor pruning selections. Extensive results on ImageNet and CIFAR datasets show consistent improvements over the prior art by large margins, for varying networks towards both structured and unstructured sparsity. For example, our method surpasses recent work by $+1.3\%$ top-1 accuracy at the same compression ratio for ResNet50-ImageNet unstructured sparsity. Moreover, our structured sparsity results also improve upon the previous best hardware-aware pruning method by $+0.8\%$ top-1 accuracy for MobileNet-ImageNet sparsification, offering $+134$ in hardware FPS(im/s), while halving the training cost.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning