AffineQuant: Affine Transformation Quantization for Large Language Models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: post-training quantization, large language model, Affine Transformation
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: The significant resource requirements associated with Large-scale Language Models (LLMs) have generated considerable interest in the development of techniques aimed at compressing and accelerating neural networks.
Among these techniques, Post-Training Quantization (PTQ) has emerged as a subject of considerable interest due to its noteworthy compression efficiency and cost-effectiveness in the context of training.
Existing PTQ methods for LLMs limit the optimization scope to scaling transformations between pre- and post-quantization weights.
This constraint results in significant errors after quantization, particularly in low-bit configurations.
In this paper, we advocate for the direct optimization using equivalent Affine transformations in PTQ (AffineQuant).
This approach extends the optimization scope and thus significantly minimizing quantization errors.
Additionally, by employing the corresponding inverse matrix, we can ensure equivalence between the pre- and post-quantization outputs of PTQ, thereby maintaining its efficiency and generalization capabilities.
To ensure the invertibility of the transformation during optimization, we further introduce a gradual mask optimization method.
This method initially focuses on optimizing the diagonal elements and gradually extends to the other elements.
Such an approach aligns with the Levy-Desplanques theorem, theoretically ensuring invertibility of the transformation.
As a result, significant performance improvements are evident across different LLMs on diverse datasets.
Notably, these improvements are most pronounced when using very low-bit quantization, enabling the deployment of large models on edge devices.
To illustrate, we attain a C4 perplexity of $15.76$ (2.26$\downarrow$ vs $18.02$ in OmniQuant) on the LLaMA2-$7$B model of W$4$A$4$ quantization without overhead.
On zero-shot tasks, AffineQuant achieves an average of $58.61\%$ accuracy ( $1.98\%\uparrow$ vs $56.63$ in OmniQuant) when using $4$/$4$-bit quantization for LLaMA-$30$B, which setting a new state-of-the-art benchmark for PTQ in LLMs.
Codes are available at: https://github.com/bytedance/AffineQuant.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
Supplementary Material: zip
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Primary Area: general machine learning (i.e., none of the above)
Submission Number: 1842
Loading