Enhancing Label-Free Cross-Task Generalization via Stability and Confidence Aware LoRA Composition

ACL ARR 2025 May Submission4358 Authors

Abstract: Recent advances in parameter-efficient fine-tuning have established Low-Rank Adaptation (LoRA) as a commonly used technique for adapting large language models (LLMs) to downstream tasks. Building on LoRA’s modularity and low resource requirements, composing multiple LoRA modules has emerged as a promising approach to enhance cross-task generalization. However, in label-free scenarios, two major challenges hinder effective unsupervised LoRA composition: (1) the lack of principled criteria for selecting relevant modules in the absence of task-specific information for unseen tasks, and (2) the reliance on training with task-specific examples to optimize module integration coefficients. To tackle these issues, we propose a two-stage method for stability- and confidence-aware LoRA composition, aimed at enhancing label-free cross-task generalization. In the first stage, we assess module robustness via stability analysis—introducing controlled noise during generation and identifying modules whose outputs remain confident and consistent. In the second stage, we generate pseudo-training data from selected modules and perform confidence-guided filtering to ensure high-quality supervision for model adaptation. Empirical results on seven diverse evaluation tasks demonstrate that our approach improves average ROUGE-L scores and outperforms existing label-free merging baselines on 42\% of tasks, showcasing its effectiveness in generalizing to unseen settings without labeled data. Our code is available at https://github.com/8k2aax0e/new-LoRA-Composition-method.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: NLP in resource-constrained settings

Contribution Types: Approaches to low-resource settings

Languages Studied: English

Submission Number: 4358