G-KV: Decoding-Time KV Cache Eviction with Global Attention

ACL ARR 2026 January Submission4113 Authors

Keywords: Large language model, Reasoning, Efficiency, Reinforcement Learning

Abstract: Recent reasoning large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths. KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning. However, existing methods often focus on prompt compression or token eviction with local attention score, overlooking the long-term importance of tokens. We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance. Additionally, we introduce post-training techniques, including reinforcement learning and distillation, to optimize models for compressed KV cache settings.

Paper Type: Long

Research Area: LLM Efficiency

Research Area Keywords: LLM Efficiency; NLP in resource-constrained settings

Contribution Types: Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 4113