Layer Collaborative Low-Rank Decomposition with Automatic Rank Search for LLM Compression

Zhijie Qiu, Tingyu Wu, Wei Luo, Qiuwu Chen, Yaofo Chen, Mingkui Tan

Keywords: Large Language Model, Singular Value Decomposition, Rank Search

TL;DR: We propose LC-SVD, a novel layer-collaborative SVD framework for compressing LLMs with automatic rank allocation. It jointly decomposes all layers within transformer blocks, preserving intra-block dependencies.

Abstract:

Large Language Models (LLMs) achieve strong performance but face deployment challenges due to high storage and memory costs. Low-rank approximation via Singular Value Decomposition (SVD) offers an effective compression solution. However, existing SVD-based methods typically compress each weight matrix independently in a layer-wise manner, ignoring the cross-layer interactions within transformer blocks and causing suboptimal performance. Moreover, conventional rank allocation strategies—either greedy or based on singular value decay—are often suboptimal, overlooking the varying sensitivity of different blocks to compression. To address these issue, we propose LC-SVD, a layer collaborative SVD framework with automatic rank search that enables adaptive low-rank compres-
sion of LLMs. Our approach includes: 1) block-wise collaborative decomposition jointly compresses all linear layers within a transformer block, preserving intra-block structural dependencies and reducing error accumulation. To improve rank allocation, we devise an error-driven rank search strategy that evaluates block sensitivity on calibration data and prioritizes capacity in more critical components via
candidate configuration scoring. This ensures better accuracy under fixed resource budgets. The experimental results show that LC-SVD outperforms state-of-the-art SVD-based methods, achieving lower perplexity and higher task performance.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 6575