Pruning neural networks: is it time to nip it in the bud?

Oct 10, 2018 Blind Submission readers: everyone
  • Abstract: Pruning is a popular technique for compressing a neural network: a large pre-trained network is fine-tuned while connections are successively removed. However, the value of pruning has largely evaded scrutiny. In this extended abstract, we examine residual networks obtained through Fisher-pruning and make two interesting observations. First, when time-constrained, it is better to train a simple, smaller network from scratch than prune a large network. Second, it is the architectures obtained through the pruning process --- not the learnt weights --- that prove valuable. Such architectures are powerful when trained from scratch. Furthermore, these architectures are easy to approximate without any further pruning: we can prune once and obtain a family of new, scalable network architectures for different memory requirements.
  • Keywords: pruning, architectures
  • TL;DR: Training small networks beats pruning, but pruning finds good small networks to train that are easy to copy.
0 Replies