PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modeling

TMLR Paper5500 Authors

Abstract: The growing number of Large Language Models (LLMs) with diverse capabilities and response styles provides users with a wider range of choices, which presents challenges in selecting appropriate LLMs, as user preferences vary in terms of performance, cost, and response style. Current LLM selection methods typically optimize for a single fixed objective, such as performance, cost, or a trade-off between them, and fail to learn user preferences from interaction data. To address these limitations in supporting users, we propose PersonalizedRouter, a graph-based framework that models diverse user profiles and performs personalized LLM selection by leveraging interaction data that includes task context, queries, candidate LLMs, and user decisions. To capture contextual information between user queries and optimal LLMs, PersonalizedRouter converts the interaction data into a heterogeneous graph, where the relationships between different types of nodes are represented by edges. To further assess the adaptability for multiple users, we design two strategies to simulate different user interaction data: the multi-cost-efficiency simulation strategy and the LLM-as-a-Judge strategy. The experimental results from two simulation settings demonstrate that our PersonalizedRouter outperforms existing LLM selection methods and surpasses the strongest methods by a large margin of 16.97% and 9.83%. In a larger-scale setting with more users and LLMs, it achieves at least 49.26% time cost reduction while outperforming all baselines and maintaining superior robustness. Moreover, PersonalizedRouter exhibits few-shot learning capabilities, effectively adapting to new users and new LLMs, achieving 64.81% and 85.80% of the fully trained model’s performance, respectively.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Yen-Chang_Hsu1

Submission Number: 5500