MixBin: Towards Budgeted Binarization

Udbhav Bamba, Neeraj Anand, Dilip K. Prasad, Deepak Gupta

Keywords: Budgeted Binarization, Model Compression, Efficient deep learning

TL;DR: We present $\texttt{MixBin}$, an iterative search-based strategy that constructs B2NN through optimized mixing of the binary and full-precision components.

Abstract: Binarization has proven to be amongst the most effective ways of neural network compression, reducing the FLOPs of the original model by a large extent. However, such levels of compression are often accompanied by a significant drop in the performance of the model. There exist some approaches that reduce this performance drop by facilitating partial binarization of the network, however, a systematic approach to mix binary and full-precision parameters in a single network is still missing. In this paper, we propose a paradigm to perform partial binarization of neural networks in a controlled sense, thereby constructing budgeted binary neural network (B2NN). We present $\texttt{MixBin}$, an iterative search-based strategy that constructs B2NN through optimized mixing of the binary and full-precision components. $\texttt{MixBin}$ allows to explicitly choose the approximate fraction of the network to be kept as binary, thereby presenting the flexibility to adapt the inference cost at a prescribed budget. We demonstrate through numerical experiments that B2NNs obtained from our $\texttt{MixBin}$ strategy are significantly better than those obtained from random selection of the network layers. To perform partial binarization in an effective manner, it is important that both the full-precision as well as the binary components of the B2NN are appropriately optimized. We also demonstrate that the choice of the activation function can have a significant effect on this process, and to circumvent this issue, we present BinReLU, an integral component of $\texttt{MixBin}$, that can be used as an effective activation function for the full-precision as well as the binary components of any B2NN. Experimental investigations reveal that BinReLU outperforms the other activation functions in all possible scenarios of B2NN: zero-, partial- as well as full binarization. Finally, we demonstrate the efficacy of $\texttt{MixBin}$ on the tasks of classification and object tracking using benchmark datasets.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/mixbin-towards-budgeted-binarization/code)