Capacity of Deep Neural Networks under Parameter Quantization

Yoonho Boo, Sungho Shin, and Wonyong Sung

Abstract: Most deep neural networks (DNNs) require complex models to achieve high performance. Parameter quantization is widely used for reducing the implementation complexities. Previous studies on quantization were mostly based on extensive simulation using training data. We choose a different approach and attempt to measure the per-parameter capacity of DNN models and interpret the results to obtain insights on optimum quantization of parameters. This research uses artificially generated data and generic forms of fully connected DNNs, convolutional neural networks, and recurrent neural networks. We conduct memorization and classification tests to study the effects of the number and precision of the parameters on the performance. The model and the per-parameter capacities are assessed by measuring the mutual information between the input and the classified output. We also extend the memorization capacity measurement results to image classification and language modeling tasks. To get insight for parameter quantization when performing real tasks, the training and test performances are compared.

Keywords: quantization, network capacity, hardware implementation, network compression

TL;DR: We suggest the sufficient number of bits for representing weights of DNNs and the optimum bits are conservative when solving real problems.