Pruning with Output Error Minimization for Producing Efficient Neural Networks

Koji Kamma, Toshikazu Wada

Keywords: pruning, weighted least squares method, convolutional neural networks, compression

TL;DR: We present a pruning method that conducts pruning and then performs "reconstruction" to minimize the output error of the activation function, while previous methods minimize the error of the value before passing through the activation function.

Abstract: DNNs are known to have excessive parameters and thus are computationally expensive, which poses a challenge for implementations in various applications. Structured pruning is a technique of compressing a trained DNN model by removing redundant neurons (or channels). How well a pruned model maintains its accuracy depends on two factors. The first is compression ratio optimization, in other words, how many neurons are reduced in each layer. The other is layer-wise optimization, in other words, which neurons to be preserved in each layer. In this paper, we propose Pruning with Output Error Minimization (POEM), a layer-wise pruning method that conducts pruning and then performs reconstruction to compensate the error caused by pruning. The strength of POEM lies in its reconstruction using the weighted least squares method so as to minimize the output error of the activation function, while the previous methods minimize the error of the value before applying the activation function. The experiments with well-known DNN models and a large scale image recognition dataset show that POEM is better than the previous methods in maintaining the accuracy of those models.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning