Align, Don’t Divide: Revisiting the LoRA Architecture in Multi-Task Learning

Jinda Liu, Bo Cheng, Yi Chang, Yuan Wu

Keywords: Large Language Models, Low-rank Adaptation, Multi-Task Learning

Abstract: Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting Large Language Models (LLMs). In practice, LLMs are often required to handle a diverse set of tasks from multiple domains, a scenario naturally addressed by multi-task learning (MTL). Within this MTL context, a prevailing trend involves LoRA variants with multiple adapters or heads, which advocate for structural diversity to capture task-specific knowledge. Our findings present a direct challenge to this paradigm. We first show that a simplified multi-head architecture with high inter-head similarity substantially outperforms complex multi-adapter and multi-head systems. This leads us to question the multi-component paradigm itself, and we further demonstrate that a standard single-adapter LoRA, with a sufficiently increased rank, also achieves highly competitive performance. These results lead us to a new hypothesis: learning task-shared representations provides a highly effective and promising path towards multi-task learning, offering a powerful alternative to the architectural isolation of task-specific features. To validate this, we propose Align-LoRA, which incorporates an explicit loss to align task representations within the shared adapter space. Theoretical analysis and experiments confirm that Align-LoRA significantly surpasses baselines, establishing a simpler yet more effective paradigm for adapting LLMs to multiple tasks. The code is available anonymously.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 9340