Structured Pruning of CNNs at Initialization

Yaohui Cai, Weizhe Hua, Hongzheng Chen, G. Edward Suh, Christopher De Sa, Zhiru Zhang

Keywords: Pruning, Pruning-at-Initialization, Structured Pruning, Efficient Deep Learning, Efficient Model, Acceleration, Deep Learning Acceleration, Convolutional Neural Network

TL;DR: Instant structured pruning-at-initialization method can be as good as unstructured ones.

Abstract: Pruning-at-initialization (PAI) methods prune the individual weights of a convolutional neural network (CNN) before training, thus avoiding expensive fine-tuning or retraining of the pruned model. While PAI shows promising results in reducing model size, the pruned model still requires unstructured sparse matrix computation, making it difficult to achieve a real speedup. In this work, we show both theoretically and empirically that the accuracy of CNN models pruned by a PAI method is independent of the granularity of pruning when layer-wise density (i.e., the fraction of the remaining parameters in each layer) remains the same. We formulate PAI as a convex optimization problem based on an expectation-based proxy for model accuracy, which can instantly produce the optimal allocation of the layer-wise densities with respect to the proxy model. Using our formulation, we further propose a structured and hardware-friendly PAI method, named PreCrop, to prune or reconfigure CNNs in the channel dimension. Our empirical results show that PreCrop achieves a higher accuracy than existing PAI methods on several popular CNN architectures, including ResNet, MobileNetV2, and EfficientNet, on both CIFAR-10 and ImageNet. Notably, PreCrop achieves an accuracy improvement of up to 1.9% over a state-of-the-art PAI algorithm when pruning MobileNetV2 on ImageNet.

Primary Area: infrastructure, software libraries, hardware, etc.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3967