Rotation Invariant Quantization for Model Compression
Keywords: Neural Network, Model compression, Rate-distortion, Quantization
TL;DR: In this study, we investigate the theoretical limits of post-training NN model compression rates using the rate-distortion theory, proving that the highest compression rate is attained by a simple single-letter (scalar) rotation-invariant solution.
Abstract: Post-training Neural Network (NN) model compression is an attractive approach for deploying large, memory-consuming models on devices with limited memory resources. In this study, we investigate the theoretical limits of the NN model compression rates using the rate-distortion theory. First, we prove that the highest compression is attained by a simple single-letter (scalar) rotation-invariant solution. Then, based on these insights, we suggest a Rotation-Invariant Quantization (RIQ) technique that finds the optimal single-letter solution efficiently, yielding a different rate at each layer, i.e., mixed-precision quantization. We rigorously evaluate RIQ and demonstrate its capabilities on various models and tasks. For example, RIQ facilitates $\times 19.4$ and $\times 52.9$ compression ratio on pre-trained VGG dense and pruned models, respectively, with $<0.4\%$ degradation in accuracy.
Code is available in the supplementary material.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Submission Guidelines: Yes
Please Choose The Closest Area That Your Submission Falls Into: Optimization (eg, convex and non-convex optimization)
Supplementary Material: zip
Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2303.03106/code)
10 Replies
Loading