AdaCache: Adaptive Caching and Context Augmentation for Efficient LLM Serving

ICLR 2026 Conference Submission20010 Authors

Keywords: LLM Inference, Retrieval-Augmented Generation, Caching, Recomputation

Abstract:

Retrieval-Augmented Generation (RAG) significantly enhances Large Language Models by integrating external knowledge sources, but at the cost of substantial computational overhead from extended input sequences.
Current RAG systems exhibit two fundamental inefficiencies: redundant processing of frequently retrieved text chunks across multiple queries, and uniform deep retrieval that over-provisions context regardless of query complexity.
We present AdaCache, an adaptive caching framework that addresses these limitations through dual optimization strategies.
First, we introduce a cache-aware partial recomputation mechanism that profiles attention patterns to construct selective cache variants, enabling flexible reuse while preserving cross-chunk dependencies.
Second, we develop adaptive context augmentation that dynamically determines optimal retrieval depth via lightweight confidence estimation, avoiding unnecessary overhead on simple queries.
Comprehensive experiments across diverse datasets and LLMs demonstrate that AdaCache delivers substantial improvements in Time-To-First-Token compared to state-of-the-art RAG caching systems, while preserving generation quality.

Primary Area: infrastructure, software libraries, hardware, systems, etc.

Submission Number: 20010