On The Energy Statistics of Feature Maps in Pruning of Neural Networks with Skip-Connections

Mohammadreza Soltani, Suya Wu, Yuerong Li, Jie Ding, Vahid Tarokh

Abstract: We propose a new structured pruning framework for compressing Deep Neural Networks (DNNs) with skip-connections, based on measuring the statistical dependency of hidden layers and predicted outputs. The dependence measure defined by the energy statistics of hidden layers serves as a model-free measure of information between the feature maps and the output of the network. The estimated dependence measure is subsequently used to prune a collection of redundant and uninformative layers. Extensive numerical experiments on various architectures show the efficacy of the proposed pruning approach with competitive performance to state-of-the-art methods.