CARROT: A Cost Aware Rate Optimal Router

ICLR 2026 Conference Submission12731 Authors

Keywords: LLM Routing, minimax rate

TL;DR: We present a theoretical and empirical investigation of performance and cost trade-offs in LLM routing.

Abstract: With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lowerbound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models’ cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard- v2 we empirically validate CARROT’s performance against several alternative routers.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 12731