iRDA Method for Sparse Convolutional Neural Networks

Xiaodong Jia, Liang Zhao, Lian Zhang, Juncai He, Jinchao Xu

Abstract: We propose a new approach, known as the iterative regularized dual averaging (iRDA), to improve the efficiency of convolutional neural networks (CNN) by significantly reducing the redundancy of the model without reducing its accuracy. The method has been tested for various data sets, and proven to be significantly more efficient than most existing compressing techniques in the deep learning literature. For many popular data sets such as MNIST and CIFAR-10, more than 95% of the weights can be zeroed out without losing accuracy. In particular, we are able to make ResNet18 with 95% sparsity to have an accuracy that is comparable to that of a much larger model ResNet50 with the best 60% sparsity as reported in the literature.

Keywords: sparse convolutional neural networks, regularized dual averaging

TL;DR: A sparse optimization algorithm for deep CNN models.