Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win

Utku Evci, Yani Ioannou, Cem Keskin, Yann Dauphin

Keywords: sparse training, sparsity, pruning, lottery ticket hypothesis, lottery tickets, sparse initialization, initialization, deep learning, gradient flow

Abstract: Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction of the compute/storage for inference, and also have the potential to enable efficient training. However, naively training unstructured sparse NNs from random initialization results in significantly worse generalization, with the notable exception of Lottery Tickets (LTs) and Dynamic Sparse Training (DST). In this work, we attempt to answer: (1) why training unstructured sparse networks from random initialization performs poorly and; and (2) what makes LTs and DST the exceptions? We show that sparse NNs have poor gradient flow at initialization and propose a modified initialization for unstructured connectivity. Furthermore, we find that DST methods significantly improve gradient flow during training over traditional sparse training methods. Finally, we show that LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from — however, this comes at the cost of learning novel solutions.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We show that sparse NNs have poor gradient flow and addressing this issue brings promising results, whereas lottery tickets seem to bypass this issue by effectively relearning the pruning solution.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.03533/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=KXUhtoB1gY