Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents

ICLR 2026 Conference Submission13016 Authors

Keywords: Large Language Models, Reasoning, Agents, System Efficiency, Information Retrieval

TL;DR: This paper demystifies the key factors affecting the efficiency of LLM-based search agents and, based on these insights, designs SearchAgent-X to improve end-to-end efficiency without compromising generation quality.

Abstract: Large Language Model (LLM)-based search agents have shown remarkable capabilities in solving complex tasks by dynamically decomposing problems and addressing them through interleaved reasoning and retrieval. However, this interleaved paradigm introduces substantial efficiency bottlenecks. First, we observe that both highly accurate and overly approximate retrieval methods degrade system efficiency: exact search incurs significant retrieval overhead, while coarse retrieval requires additional reasoning steps during generation. Second, we identify inefficiencies in system design, including improper scheduling and frequent retrieval stalls, which lead to cascading latency---where even minor delays in retrieval amplify end-to-end inference time. To address these challenges, we introduce \texttt{SearchAgent-X}, a high-efficiency inference framework for LLM-based search agents. \texttt{SearchAgent-X} leverages high-recall approximate retrieval and incorporates two key techniques: priority-aware scheduling and non-stall retrieval. Extensive experiments demonstrate that \texttt{SearchAgent-X} consistently outperforms state-of-the-art systems such as vLLM and HNSW-based retrieval across diverse tasks, achieving up to 3.4$\times$ higher throughput and 5$\times$ lower latency, without compromising generation quality. Code is available at \url{https://anonymous.4open.science/r/SearchAgent-X}.

Primary Area: infrastructure, software libraries, hardware, systems, etc.

Submission Number: 13016