Sparse Weight Activation Training

Md Aamir Raihan, Tor M. Aamodt

Keywords: Sparsity, Training, Acceleration, Pruning, Compression

TL;DR: We sparsified computation by up to 70-80% in both forward and backward passes during CNN training with minimal loss in accuracy.

Abstract: Training convolutional neural networks (CNNs) is time consuming. Prior work has explored how to reduce the computational demands of training by eliminating gradients with relatively small magnitude. We show that eliminating small magnitude components has limited impact on the direction of high-dimensional vectors. However, in the context of training a CNN, we find that eliminating small magnitude components of weight and activation vectors allows us to train deeper networks on more complex datasets versus eliminating small magnitude components of gradients. We propose Sparse Weight Activation Training (SWAT), an algorithm that embodies these observations. SWAT reduces computations by 50% to 80% with better accuracy at a given level of sparsity versus the Dynamic Sparse Graph algorithm. SWAT also reduces memory footprint by 23% to 37% for activations and 50% to 80% for weights.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2001.01969/code)