Rigging the Lottery: Making All Tickets Winners

Utku Evci, Erich Elsen, Pablo Castro, Trevor Gale

TL;DR: We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNet-2012 dataset.

Abstract: Sparse neural networks have been shown to yield computationally efficient networks with improved inference times. There is a large body of work on training dense networks to yield sparse networks for inference (Molchanov et al., 2017;Zhu & Gupta, 2018; Louizos et al., 2017; Li et al., 2016; Guo et al., 2016). This limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods. Our method updates the topology of the network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires less floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results with ResNet-50, MobileNet v1 and MobileNet v2 on the ImageNet-2012 dataset. Finally, we provide some insights into why allowing the topology to change during the optimization can overcome local minima encountered when the topology remains static.

Code: https://drive.google.com/file/d/1XdexLVd2_PkgUu8mjkjKQpA2zvsaiqKe/view?usp=sharing

Keywords: sparse training, sparsity, pruning, lottery tickets, imagenet, resnet, mobilenet, efficiency, optimization, local minima

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1911.11134/code)