KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs

Jian Chen, Zhuoran Wang, Jiayu Qin, Ming Li, Meng Wang, Changyou Chen, Qizhen Weng, Yin Chen, Yirui Liu

Keywords: KV-cache, Large Language Models, Model Compression, Low-Rank Analysis

TL;DR: KV-CORE is an efficient method for computing SVD of dataset-level KV-cache sequences in LLMs, enabling data-dependent compressibility evaluation and yielding optimal low-rank compression projections.

Abstract: Large language models rely on kv-caches to avoid redundant computation during autoregressive decoding, but as context length grows, reading and writing the cache can quickly saturate GPU memory bandwidth. Recent work has explored KV-cache compression, yet most approaches neglect the data-dependent nature of kv-caches and their variation across layers. We introduce \textbf{KV-CoRE} (\textbf{KV}-cache \textbf{Co}mpressibility by \textbf{R}ank \textbf{E}valuation), an SVD-based method for quantifying the data-dependent low-rank compressibility of kv-caches. KV-CoRE computes the optimal low-rank approximation under the Frobenius norm and, being gradient-free and incremental, enables efficient dataset-level, layer-wise evaluation. Using this method, we analyze multiple models and datasets spanning five English domains and sixteen languages, uncovering systematic patterns that link compressibility to model architecture, training data, and language coverage. As part of this analysis, we employ the Normalized Effective Rank as a metric of compressibility and show that it correlates strongly with performance degradation under compression. Our study establishes a principled evaluation framework and the first large-scale benchmark of kv-cache compressibility in LLMs, offering insights for dynamic, data-aware compression and data-centric model development.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 4831