Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization

Bharathwaj Krishnaswami Sreedhar, Vinit Jain

Abstract: Binary neural networks, until now have been optimized using the existing optimizers such as Adam or SGD. However, the existing optimizers are built for latent weights and do not perform well when used on Binarized Neural Networks. The paper proposes a novel "Binary Optimizer" ($BoP$) specifically developed for binary neural networks. A VGG like network is trained on the CIFAR-10 dataset, and finally compared against the baseline model. To conclude, rigorous experiments are performed by varying the hyper-parameters, and evaluating on different data-sets and architectures, thereby evaluating the overall training of the Binary Neural Networks against the conventional CNNs.

Track: Baseline

NeurIPS Paper Id: https://openreview.net/forum?id=BJe_nNrgIS