Quantization vs Pruning: Insights from the Strong Lottery Ticket Hypothesis

Aakash Kumar, Emanuele Natale

Keywords: Neural Network Pruning, Strong Lottery Ticket Hypothesis, Quantization, Number Partitioning Problem, Random Subset Sum Problem

TL;DR: We provide optimal relationships between the network size and weight precision of a large network such that it can be pruned to a given target network.

Abstract:

Quantization is an essential technique for making neural networks more efficient, yet our theoretical understanding of it remains limited. Previous works demonstrated that extremely low-precision networks, such as binary networks, can be constructed by pruning large, randomly-initialized networks, and showed that the ratio between the size of the original and the pruned networks is at most polylogarithmic.
The specific pruning method they employed inspired a line of theoretical work known as the Strong Lottery Ticket Hypothesis (SLTH), which leverages insights from the Random Subset Sum Problem. However, these results primarily address the continuous setting and cannot be applied to extend SLTH results to the quantized setting.
In this work, we build on foundational results by Borgs et al. on the Number Partitioning Problem to derive new theoretical results for the Random Subset Sum Problem in a quantized setting.
Using these results, we then extend the SLTH framework to finite-precision networks. While prior work on SLTH showed that pruning allows approximation of a certain class of neural networks, we demonstrate that, in the quantized setting, the analogous class of target discrete neural networks can be represented exactly, and we prove optimal bounds on the necessary overparameterization of the initial network as a function of the precision of the target network.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 16573