Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives

Elliot Meyerson, Xin Qiu

TL;DR: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives

Abstract: Decomposing hard problems into subproblems often makes them easier and more efficient to solve. With the high cost of running LLMs at scale, there is an increasing effort to decompose systems into sets of LLM-based agents, each of whom can be delegated sub-tasks. However, this decomposition (even when automated) is often intuitive, e.g., based on how a human might assign roles to members of a human team. How close are these role decompositions to optimal? This position paper argues that asymptotic analysis with LLM primitives is needed to reason about the efficiency of such problem decompositions, and that insights from such analysis will unlock opportunities for scaling such systems. By treating the LLM forward pass as the atomic unit of computational cost, one can separate out the (often opaque) inner workings of a particular LLM from the inherent efficiency of how a set of LLMs are orchestrated to solve hard problems. In other words, if we want to scale the deployment of LLMs to the limit, instead of anthropomorphizing LLMs, asymptotic analysis with LLM primitives should be used to reason about and develop more powerful decompositions of large problems into LLM agents.

Lay Summary:

It has become popular to use large language models to build "agents". Each agent is a program with a specific role to play in the world. To solve complex problems, it is natural to build a system with multiple agents, each assigned a complementary role. To date, such agent-based systems have been organized in an intuitive or ad-hoc manner, for example, assigning roles to agents in the same way we would assign jobs to humans in an organization. How close are such agent organizations to optimal? It turns out they are not close at all!

This paper proposes a way to analyze the scalability of such systems, grounded in the classical computer science tool of algorithm analysis. Through a series of examples, we use this approach to show that an intuitive agent-based system can be drastically improved by more carefully considering the algorithmic structure of the problem. This analysis tool will thus help people build agent-based systems that are far more efficient and scalable, and the organization of such future systems will not be humanlike, but uncompromisingly machinelike.

Primary Area: Research Priorities, Methodology, and Evaluation

Keywords: Large language models, Agents, Analysis of Algorithms, Decomposition, Anthropomorphism, LLM Primitives, Asymptotic Analysis

Submission Number: 315