Recency/Frequency Adaptive KV Caching for Large Language Model Serving

Yang Shen, Meghana Madhyastha, Robert Underwood, Bogdan Nicolae, Randal Burns

Keywords: LLM Serving, KV Cache, Workload-Aware Caching

Abstract: Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diverse, which makes them difficult to cache effectively. Existing cache management strategies adopt the least-recently-used policy for evicting cache blocks. However, LRU leads to multiple unrelated workloads flushing each other's caches. To address this, we integrate adaptive caching that dynamically allocates cache space between recently and frequently occurring KV blocks. Evaluations show that it improves the KV cache hit rate by up to 10.8\% and reduces time to first token by up to 12.6\% over naive vLLM on synthetic document question answering workloads, and 2.1\% and 2.0\% respectively on real-world conversation workloads. The method generalizes well to batch inference and demonstrates clear interpretability while effectively accommodating diverse workloads.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 9