The Pitfalls of KV Cache Compression

ICLR 2026 Conference Submission22530 Authors

Keywords: KV Cache Compression, Large Language Model, Inference, System Prompt

TL;DR: We study the effects of KV cache compression under a multi-instruction setting, showing that some instructions are ignored when compressed. We then argue that this phenomenon can be mitigated by reducing the eviction bias towards certain instructions

Abstract: KV cache compression promises increased throughput and efficiency with negligible loss in performance. While the gains in throughput are indisputable and recent literature has indeed shown minimal degradation on particular benchmarks, in general the consequences of compression in realistic scenarios such as multi-instruction prompting have been insufficiently studied. In this paper, we identify several pitfalls practitioners should be aware of when deploying KV cache compressed LLMs. Importantly, we show that certain instructions degrade much more rapidly with compression, effectively causing them to be completely ignored by the LLM. As a practical example of that, we highlight system prompt leakage as a case study, empirically showing the impact of compression on leakage and general instruction following. We show several factors that play a role in prompt leakage: compression method, instruction order, and KV eviction bias. We then propose simple changes to KV cache eviction policies that can reduce the impact of these factors and improve the overall performance in multi-instruction tasks.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 22530