Discrete-Valued Neural Networks Using Variational Inference

Wolfgang Roth, Franz Pernkopf

Abstract: The increasing demand for neural networks (NNs) being employed on embedded devices has led to plenty of research investigating methods for training low precision NNs. While most methods involve a quantization step, we propose a principled Bayesian approach where we first infer a distribution over a discrete weight space from which we subsequently derive hardware-friendly low precision NNs. To this end, we introduce a probabilistic forward pass to approximate the intractable variational objective that allows us to optimize over discrete-valued weight distributions for NNs with sign activation functions. In our experiments, we show that our model achieves state of the art performance on several real world data sets. In addition, the resulting models exhibit a substantial amount of sparsity that can be utilized to further reduce the computational costs for inference.

TL;DR: Variational Inference for infering a discrete distribution from which a low-precision neural network is derived

Keywords: low-precision, neural networks, resource efficient, variational inference, Bayesian

Data: [MNIST](https://paperswithcode.com/dataset/mnist)