Random Partition Relaxation for Training Binary and Ternary Weight Neural Network

Lukas Cavigelli, Luca Benini

Keywords: binary weight neural networks, ternary weight neural networks, quantization, quantized neural networks

TL;DR: State-of-the-art training method for binary and ternary weight networks based on alternating optimization of randomly relaxed weight partitions

Abstract: We present Random Partition Relaxation (RPR), a method for strong quantization of the parameters of convolutional neural networks to binary (+1/-1) and ternary (+1/0/-1) values. Starting from a pretrained model, we first quantize the weights and then relax random partitions of them to their continuous values for retraining before quantizing them again and switching to another weight partition for further adaptation. We empirically evaluate the performance of RPR with ResNet-18, ResNet-50 and GoogLeNet on the ImageNet classification task for binary and ternary weight networks. We show accuracies beyond the state-of-the-art for binary- and ternary-weight GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using a SGD-based training method that can easily be integrated into existing frameworks.