Lookahead: A Far-sighted Alternative of Magnitude-based Pruning


Sep 25, 2019 Blind Submission readers: everyone Show Bibtex
  • TL;DR: We study a multi-layer generalization of the magnitude-based pruning.
  • Abstract: Magnitude-based pruning is one of the simplest methods for pruning neural networks. Despite its simplicity, magnitude-based pruning and its variants have shown state-of-the-art performances for pruning modern architectures. Based on the observation that the magnitude-based pruning indeed minimizes the Frobenius distortion of a linear operator corresponding to a single layer, we develop a simple pruning method, coined lookahead pruning, by extending the single layer optimization to a multi-layer optimization. Our experimental results demonstrate that the proposed method consistently outperforms the magnitude pruning on various networks including VGG and ResNet, particularly in the high-sparsity regime.
  • Keywords: network magnitude-based pruning
0 Replies