Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Exploring Sparsity in Recurrent Neural Networks
Sharan Narang, Greg Diamos, Shubho Sengupta, Erich Elsen
Nov 04, 2016 (modified: Feb 22, 2017)ICLR 2017 conference submissionreaders: everyone
Abstract:Recurrent neural networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8× and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse GEMMs. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2× to 7×.
TL;DR:Reduce parameter count in recurrent neural networks to create smaller models for faster deployment
Keywords:Speech, Deep learning, Supervised Learning
Enter your feedback below and we'll get back to you as soon as possible.