Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training

Aleksandra Nowak, Bram Grooten, Decebal Constantin Mocanu, Jacek Tabor

Keywords: Dynamic Sparse Training, Pruning, Deep Learning

TL;DR: We study the impact of different pruning criteria on the performance of Dynamic Sparse Training.

Abstract: Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network’s sparse connectivity. While the growing criterion's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue, we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude-based pruning.

Submission Number: 8211