Go to ICLR 2026 Conference homepageSoCo: Progressive Spectrum Optimization for Large Language Model Compression
Keywords: Large Language Model, Model Compression, Singular Value Decomposition, Data‑Driven Optimization, Low‑Rank Approximation, Singular Spectrum Rescaling
TL;DR: We propose SoCo, a data‑driven SVD‑based compression framework that learns to rescale and sparsify the singular spectrum via a three‑stage optimization, yielding superior compression–performance trade‑offs for large language models.
Abstract: Large language models (LLMs) have demonstrated remarkable capabilities, yet prohibitive parameter complexity often hinders their deployment. Existing singular value decomposition (SVD) based compression methods equate singular values with component importance, an assumption that often fails to correlate with downstream task performance. In this work, we introduce SoCo (Singular spectrum optimization for large language model compression), a novel framework that learns to rescale SVD components. Concretely, we employ a learnable diagonal matrix to assign importance scores and introduce Progressive Spectrum Optimization, a principled strategy that operates in a single, continuous training run. Inspired by heuristic optimization, this process guides the learnable scores through distinct functional phases—from an initial exploration of the solution space, through an oscillation refinement, to a final, decisive sparsification—thereby navigating the complex optimization landscape to balance compression and performance. Thanks to this adaptive process, SoCo prunes components based on their learned importance, rather than a fixed order. More importantly, amplified scores on preserved components allow them to compensate for the information loss from pruning. Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses state-of-the-art methods in large language model compression.
Primary Area: foundation or frontier models, including LLMs
Submission Number: 4135
Loading