Incremental Soft Pruning to Get the Sparse Neural Network During Training

Kehan Zhu, Fuyi Hu, Yuanbin Ding, Yunyun Dong, Ruxin Wang

Abstract: The traditional three-stage pruning pipeline is first to train an original dense network, then identify redundant parts of the network for pruning based on the evaluation metrics of the pruning algorithm, and finally fine-tuning the pruned model, which is a time-consuming and computationally expensive process. Traditional pruning algorithms are greedy and aggressive, which may cause many important network connections to be pruned incorrectly, resulting in significant performance degradation. In this paper, we propose an incremental soft pruning during training method with the following characteristics: 1) Given the pruning rate of the network, a trained sub-network, which has performance comparable to the original network, can be obtained after training. 2) We propose three incremental pruning rate growth functions and allow the network structure to be dynamically adjusted during training to avoid pruning important network connections. 3) During the network training process, we also introduce gradient restriction, which only updates important network connections to discover good sub-network structures better. Extensive experiments show that our method can achieve better results than previous methods in different datasets and network models.