Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective

ICLR 2026 Conference Submission17242 Authors

Keywords: Efficient AI, Large Language Model; LLM Inference

TL;DR: Select critical KV cache based on constraining the output perturbation

Abstract:

Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's reliance on self-attention, particularly the large Key-Value (KV) cache for long-sequence inference.
Recent efforts to reduce KV cache size by pruning less critical entries based on attention weights remain empirical and lack formal grounding. This paper presents a formal study on identifying critical KV cache entries by analyzing attention output perturbation.
Our analysis reveals that, beyond attention weights, the value states within KV entries and pretrained parameter matrices are also crucial.
Based on this, we propose a perturbation-constrained selection algorithm that optimizes the worst-case output perturbation to identify critical entries. We demonstrate that our algorithm is a universal, plug-and-play enhancement that incurs negligible computational overhead. When integrated with three state-of-the-art cache eviction methods on three distinct LLMs, our algorithm significantly reduces the compression loss by more than half on average across 29 datasets from the Ruler and LongBench benchmarks. Further perturbation analysis, at both the head and layer levels, confirms the principles underlying our effectiveness. This work offers a new, formally grounded perspective to the cache eviction field, opening promising avenues for future research.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 17242