Sensitivity pruner: Filter-Level compression algorithm for deep neural networks

Suhan Guo, Bilan Lai, Suorong Yang, Jian Zhao, Furao Shen

Abstract: Highlights•We integrate the sensitivity measure from SNIP into the “training while fine-tuning” framework to form a more powerful pruning strategy by adapting the unstructured pruning measure from SNIP to allow filterlevel compression. In practice, the sensitivity score can be easily computed as the gradient of the connection mask applied to the weight matrix. Independent of the model structure, the sensitivity score can be applied to most neural networks for pruning purposes.•We mitigate the sampling bias in the single-shot influence score by introducing the difference between the learned pruning strategy and the single-shot strategy as the second loss component. Filter influence is measured on batched data, where a convolutional layer is used to discover the robust influence from the noise of the batch. The learning process is guided by the score provided by the influence measure.•Our algorithm can dynamically tweak the training goal between improving model accuracy and pruning more filters. We add a selfadaptive hyper-parameter