LoRA on the Go: Instance-level Dynamic LoRA Selection and Merging

ACL ARR 2026 January Submission3985 Authors

Keywords: Adapter Merging, Training-free, Instance-specific

Abstract:

Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for fine-tuning large language models.
However, conventional LoRA adapters are typically trained for a single task, limiting their applicability in real-world settings, where inputs may span multiple, diverse task domains. At inference time, existing methods can combine multiple LoRAs to improve cross-task performance, but they require additional labeled data or task-specific training, which is expensive at scale.

In this work, we introduce LoRA on the Go (LoGo), a training-free framework that dynamically selects and merges adapters at the instance level without any additional requirements. LoGo leverages signals extracted from a single forward pass through LoRA adapters, to identify the most relevant adapters and determine their contributions on-the-fly. Across 5 NLP benchmarks, 27 datasets, and 3 model families, LoGo outperforms training-based baselines on some tasks upto a margin of 3.6% while remaining competitive on other tasks and maintaining inference throughput, highlighting its effectiveness and practicality.

Paper Type: Long

Research Area: Natural Language Generation

Research Area Keywords: inference methods,domain adaptation

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 3985