Instant Quantization of Neural Networks using Monte Carlo Methods

Gonçalo Mordido, Matthijs Van Keirsbilck, Alexander Keller

TL;DR: Monte Carlo methods for quantizing pre-trained models without any additional training.

Abstract: Low bit-width integer weights and activations are very important for efficient inference, especially with respect to lower power consumption. We propose to apply Monte Carlo methods and importance sampling to sparsify and quantize pre-trained neural networks without any retraining. We obtain sparse, low bit-width integer representations that approximate the full precision weights and activations. The precision, sparsity, and complexity are easily configurable by the amount of sampling performed. Our approach, called Monte Carlo Quantization (MCQ), is linear in both time and space, while the resulting quantized sparse networks show minimal accuracy loss compared to the original full-precision networks. Our method either outperforms or achieves results competitive with methods that do require additional training on a variety of challenging tasks.

Keywords: monte carlo, importance sampling, network quantization

Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/arxiv:1905.12253/code)