A Gradient Flow Framework For Analyzing Network Pruning

Ekdeep Singh Lubana, Robert P. Dick

Keywords: Network pruning, Gradient flow, Early pruning

Abstract: Recent network pruning methods focus on pruning models early-on in training. To estimate the impact of removing a parameter, these methods use importance measures that were originally designed to prune trained models. Despite lacking justification for their use early-on in training, such measures result in surprisingly low accuracy loss. To better explain this behavior, we develop a general framework that uses gradient flow to unify state-of-the-art importance measures through the norm of model parameters. We use this framework to determine the relationship between pruning measures and evolution of model parameters, establishing several results related to pruning models early-on in training: (i) magnitude-based pruning removes parameters that contribute least to reduction in loss, resulting in models that converge faster than magnitude-agnostic methods; (ii) loss-preservation based pruning preserves first-order model evolution dynamics and its use is therefore justified for pruning minimally trained models; and (iii) gradient-norm based pruning affects second-order model evolution dynamics, such that increasing gradient norm via pruning can produce poorly performing models. We validate our claims on several VGG-13, MobileNet-V1, and ResNet-56 models trained on CIFAR-10/CIFAR-100.

One-sentence Summary: This paper establishes the relationship between regularly used importance measures for network pruning and evolution of model parameters under gradient flow, thus providing useful insights into pruning early-on in training.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ EkdeepSLubana/flowandprune](https://github.com/EkdeepSLubana/flowandprune)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2009.11839/code)