SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
Keywords: Large Language Model; Post-training Model Compression
TL;DR: We design a SVD-based post-training method to compress large language models
Abstract: The advancements in Large Language Models (LLMs) have been hindered by
their substantial sizes, which necessitates LLM compression methods for practical
deployment. Singular Value Decomposition (SVD) offers a promising solution for
LLM compression. However, state-of-the-art SVD-based LLM compression meth-
ods have two key limitations: truncating smaller singular values may lead to higher
compression loss, and the lack of update on the compressed weights after SVD
truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM
compression method that addresses the limitations of existing methods. SVD-LLM
incorporates a truncation-aware data whitening technique to ensure a direct map-
ping between singular values and compression loss. Moreover, SVD-LLM adopts
a parameter update with sequential low-rank approximation to compensate for
the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10
datasets and seven models from three different LLM families at three different
scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts,
especially at high model compression ratios.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2301
Loading