Towards the Dynamics of Representation Changes during Parameter Pruning for Network Compression

Fangbo Li, Junpeng Zhang, Qihan Ren, Quanshi Zhang

Keywords: Neural Network Compression, Parameter Pruning

TL;DR: This paper discovers a three-phase dynamics of feature representations of a neural network when we conduct parameter pruning with an increasing pruning ratio.

Abstract: This study explores the dynamics of interactions encoded by deep neural networks (DNNs) when we prune network parameters with an increasing pruning ratio. We discover a three-phase dynamics of the generalizability of the interactions removed by the parameter pruning operation, which clarifies a central issue in symbolic generalization, i.e., how interactions serve as the underlying factors that determine the change of a DNN's performance. Experimental results demonstrate that the pruning operation mainly removes high-order interactions at low pruning ratios. Because the removed high-order interactions are usually unlikely to generalize, the removal of high-order interactions has a negligible impact on testing performance. In contrast, under higher pruning ratios, both low-order and high-order interactions are gradually removed. The high generalizability of the removed low-order interactions leads to a noticeable decline in testing performance.

Primary Area: interpretability and explainable AI

Submission Number: 10430