On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks

Yukun Ding, Jinglan Liu, Jinjun Xiong, Yiyu Shi

Abstract: Compression is a key step to deploy large neural networks on resource-constrained platforms. As a popular compression technique, quantization constrains the number of distinct weight values and thus reducing the number of bits required to represent and store each weight. In this paper, we study the representation power of quantized neural networks. First, we prove the universal approximability of quantized ReLU networks on a wide class of functions. Then we provide upper bounds on the number of weights and the memory size for a given approximation error bound and the bit-width of weights for function-independent and function-dependent structures. Our results reveal that, to attain an approximation error bound of $\epsilon$, the number of weights needed by a quantized network is no more than $\mathcal{O}\left(\log^5(1/\epsilon)\right)$ times that of an unquantized network. This overhead is of much lower order than the lower bound of the number of weights needed for the error bound, supporting the empirical success of various quantization techniques. To the best of our knowledge, this is the first in-depth study on the complexity bounds of quantized neural networks.

Keywords: Quantized Neural Networks, Universial Approximability, Complexity Bounds, Optimal Bit-width

TL;DR: This paper proves the universal approximability of quantized ReLU neural networks and puts forward the complexity bound given arbitrary error.