SEQR: Secure and Efficient QR-based LoRA Routing

ICLR 2026 Conference Submission14474 Authors

Keywords: LoRA, routing, unsupervised, classification, activations

TL;DR: We formalize the goal of unsupervised LoRA routing and develop an algorithm with provable routing guarantees, better efficiency, and increased task performance over previous methods.

Abstract: Low-Rank Adaptation (LoRA) has become a standard technique for parameter-efficient fine-tuning of large language models, enabling large libraries of LoRAs, each for a specific task or domain. Efficiently selecting the correct LoRA adapter for a given input remains a challenge, particularly in secure environments where supervised training of routers may raise privacy concerns. Motivated by previous approaches, we formalize the goal of unsupervised LoRA routing in terms of activation norm maximization, providing a theoretical framework for analysis. We demonstrate the discriminative power of activation norms and introduce SEQR, an unsupervised LoRA routing algorithm designed to maximize efficiency while providing strict routing guarantees. SEQR provably identifies the norm-maximizing adapter with significantly greater efficiency, making it a highly scalable and effective solution for dynamic LoRA composition. We validate our results through experiments that demonstrate improved multi-task performance and efficiency.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 14474