Did You Check the Right Pocket? Cost-Sensitive Store Routing for Memory-Augmented Agents

Madhava Gaikwad

Keywords: memory routing, memory-augmented agents, retrieval-augmented generation, selective retrieval, context efficiency, agent memory systems, cost-sensitive retrieval, long-context reasoning, LLM systems optimization

TL;DR: Selective memory routing improves QA accuracy while reducing retrieval cost.

Abstract: Memory-augmented agents maintain multiple specialized stores, yet most systems retrieve from all stores for every query, increasing cost and introducing irrelevant context. We formulate memory retrieval as a store-routing problem and evaluate it using coverage, exact match, and token efficiency metrics. On downstream question answering, an oracle router achieves higher accuracy while using substantially fewer context tokens compared to uniform retrieval, demonstrating that selective retrieval improves both efficiency and performance. Our results show that routing decisions are a first-class component of memory-augmented agent design and motivate learned routing mechanisms for scalable multi-store systems. We additionally formalize store selection as a cost-sensitive decision problem that trades answer accuracy against retrieval cost, providing a principled interpretation of routing policies.

Submission Number: 104