An Empirical Study on Channel Pruning through Residual Connections

Jong-Ryul Lee, Yong-Hyuk Moon

Abstract: Channel pruning is one of promising techniques for compressing convolutional neural networks due to its usefulness and runtime-agnostic efficiency. Most existing works focus on determining which channel should be removed or kept for only sequentially connected convolution layers inside a residual block due to the hardness of channel pruning through residual connections. Motivated by this, we focus on investigating the layer-wise dynamics of channel pruning through residual connections, and propose simple yet effective pruning methods. These methods do not require any additional training data to compute the importance of channels affected by residual connections. In experiments, we demonstrate that the proposed methods have promising performance despite their simplicity and efficiency. In addition, we achieve interesting observations about the dynamics of channel pruning on layers at different topological positions, which are related to the consistency of layers' filters for evaluating the importance of each out-channel.