Universal Model Routing for Efficient LLM Inference

ICLR 2026 Conference Submission8068 Authors

Keywords: model routing, adaptive computation, learning to defer, efficient inference

TL;DR: We propose a principled model routing framework which allows new models to be added to or removed from the serving pool without having to retrain the routing model.

Abstract: Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose UniRoute, a new approach to this problem that relies on representing each LLM as afeature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective instantiations of UniRoute, relying on cluster-based routing and a learned cluster map respectively. We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound. Experiments on a range of public benchmarks show the effectiveness of UniRoute in routing amongst more than 30 unseen LLMs.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 8068