PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts

ICLR 2026 Conference Submission14493 Authors

Keywords: Multi-task Learning, Parameter-Efficient Fine-Tuning, Neural Architecture Engineering, Continuous Prompts Optimization, Low-Rank Adaptation

TL;DR: We prototype PEML by creating an automated framework for optimizing the continuous prompts and adapting model weights.

Abstract:

Parameter-Efficient Fine-Tuning (PEFT) is critical for adapting Large Language Models (LLMs) for various tasks.
Recently, there has been an increasing demands for fine-tuning LLMs for multiple tasks because it requires overall less data for fine-tuning thanks to the common features shared among tasks. More importantly, LLMs are resource demanding and deploying a single model for multiple tasks facilitates resource consolidation and consumes significantly less resources compared to deploying individual large model for each task. Existing PEFT methods like LoRA and Prefix Tuning are designed to adapt to a specific task. LoRA and its variation focus on aligning the model itself for tasks, overlooking the importance of prompt tuning in multi-task learning while Prefix Tuning only adopts a simple architecture to optimize prompts, which limits the adaption capabilities for multi-task. To enable efficient fine-tuning for multi-task learning, it is important to co-optimize prompt optimization and model adaptation. In this work, we propose a Parameter-Efficient Multi-task Learning (PEML), which employs a neural architecture engineering method for optimizing the continuous prompts while also performing low-rank adaption for model weights. We prototype PEML by creating an automated framework for optimizing the continuous prompts and adapting model weights. We compare against state-of-the-arts MTL-LoRA, MultiLoRa, C-Poly, and MoE, and results on the GLUE, SuperGLUE, Massive Multitask Language Understanding and commonsense reasoning benchmarks. The evaluation results presents an average accuracy improvement of up to 6.67%, with individual tasks showing peak gains of up to 10.75%.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 14493