How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities

ACL ARR 2025 May Submission5471 Authors

Abstract: Large language model (LLM) routing has emerged as a crucial strategy for balancing computational costs with performance by dynamically assigning queries to the most appropriate model based on query complexity. Despite recent advances showing that preference-data-based routers can outperform traditional methods, current evaluation benchmarks remain limited—they largely focus on general model capabilities while overlooking task-specific behaviors and critical concerns such as privacy, safety, and potential backdoor vulnerabilities introduced through preference data. In response, we propose the DSC benchmark $\textit{\textbf{D}}iverse$, $\textit{\textbf{S}}imple$, and $\textit{\textbf{C}}ategorized$, an evaluation framework that categorizes router performance across a broad spectrum of query types—including coding, translation, mathematics, human instructions, general knowledge, and LLM jailbreaking—and integrates privacy and safety assessments to reveal hidden risks. Our experiments on three preference-based routers and two commercial counterparts demonstrate that while these systems improve efficiency, they often make suboptimal, category-driven decisions; for instance, a BERT-based router directs all coding and mathematics queries to the most powerful LLM—even when simpler models would suffice—while routing jailbreaking attempts to weaker models, thereby elevating safety risks.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Safety and Alignment in LLMs

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 5471