Training Binary Neural Networks in a Binary Weight Space

Tatsukichi Shibuya, Nakamasa Inoue, Rei Kawakami, Ikuro Sato

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: binary neural network, optimization, low-precision neural network

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Propose a training algorithm for binary (1-bit) neural networks without holding any real-valued weights

Abstract:

Binary neural networks (BNNs), which have binary weights and activations, hold significant potential for enabling neural computations on low-end edge devices with limited computational power and memory resources. Currently, most existing BNN training approaches optimize and binarize real-valued weights, leading to substantial memory usage during training.Though training BNNs without real-valued weights to save memory is intriguing, it has been deemed challenging with gradient-based optimization. To address this challenge, we define an update probability for binary weights, determined by the current binary weights and real-valued gradients. The binary weights generated by our method match those obtained by SGD in the real-space training of BNNs in the expectation. As a result, the training of binary weights becomes stable even without real weights. Our method yields test results on Tiny-ImageNet comparable to baselines that utilize real weights during training, yet reduces memory usage by up to a factor of 33.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5932