Angle-DFQ: Angle aware data free quantization
Keywords: Data free quantization, Computer vision
Abstract: Data free quantization of neural networks is a practical necessity as access to training data in many situations is restricted due to privacy, proprietary concerns, or memory issues. We present a data free weight rounding algorithm for Deep Neural Networks (DNNs) that does not require any training data, synthetic data generation, fine-tuning, or even batch norm statistics. Instead, our approach focuses on preserving the direction of weight vectors during quantization. We demonstrate that traditional weight rounding techniques, that round weights to the nearest quantized level, can result in large angles between the full-precision weight vectors and the quantized weight vectors, particularly under coarse quantization regimes. For a large class of high-dimensional weight vectors in DNNs, this angle error can approach 90 degrees. By minimizing this angle error, we significantly improve top-1 accuracy in quantized DNNs. We analytically derive the angle-minimizing rounding boundaries for ternary quantization under the assumption of Gaussian weights. Building on this, we propose a greedy data-free quantization method based on the cosine similarity between the full-precision weight vectors and the quantized weight vectors. Our approach consistently outperforms existing state-of-the-art data-free quantization techniques and, in several cases, surpasses even data-dependent methods on well-established models such as ResNet-18, VGG-16, and AlexNet with aggressive quantization levels of 3 to 6 bits on the ImageNet dataset.
Primary Area: other topics in machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12754
Loading