LLM-Codebook for Extreme Compression of Large Language Models

Juncan Deng, Shuaiting Li, Chengxuan Wang, Hong Gu, Haibin Shen, Kejie Huang

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: large language model, compression, codebook, Hessian information

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Large Language Models (LLMs) have exhibited outstanding performance in both understanding and generating language. However, their remarkable abilities often correlate with large model sizes, leading to challenges during deployment, inference, and training phases. While weight quantization and pruning are prevalent strategies, they tend to lose crucial information under extreme compression.
In this paper, we propose LLM-Codebook for extreme compression of large language models (LLM-Codebook), which maps expansive LLMs (in GB) to compact codebooks (in KB). The foundation of LLM-Codebook is our novel Hessian-aware K-means algorithm, which clusters weights into codebooks based on Hessian information, preserving parameters that have significant impacts on predictions. Simultaneously, the tuning technique, LoRA is adopted to update layers that have not been compressed, aiming to recover performance using only a limited corpus. LLM-Codebook effectively preserves the generation and multi-task solving abilities of LLMs, surpassing advanced methods such as GPTQ, QLoRA, LLM-Pruner, and SparseGPT. We validate our approach by extremely compressing LLaMA-7B and Vicuna-7B to a memory requirement of 2GB (a 6x compression factor) while retaining 99% of the baseline performance. Furthermore, our approach maintains reasonable accuracy even under extreme compression ratio, achieving 90% of the original performance (36% better than GPTQ) when the model size is compressed to one-eighth.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2709