Scaling Up Exact Neural Network Compression by ReLU StabilityDownload PDF

21 May 2021, 20:48 (edited 28 Jan 2022)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: Scaling, Neural Networks, Lossless Compression, ReLU
  • TL;DR: We scale up the lossless compression in ReLU neural network based on solving a single optimization problem to identify all stable neurons.
  • Abstract: We can compress a rectifier network while exactly preserving its underlying functionality with respect to a given input domain if some of its neurons are stable. However, current approaches to determine the stability of neurons with Rectified Linear Unit (ReLU) activations require solving or finding a good approximation to multiple discrete optimization problems. In this work, we introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 183 times faster than the state-of-art method on CIFAR-10, which allows us to explore exact compression on deeper (5 x 100) and wider (2 x 800) networks within minutes. For classifiers trained under an amount of L1 regularization that does not worsen accuracy, we can remove up to 56% of the connections on the CIFAR-10 dataset. The code is available at the following link, https://github.com/yuxwind/ExactCompression .
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/yuxwind/ExactCompression
18 Replies

Loading