MT2ST: Adaptive Multi-Task to Single-Task Learning

ACL ARR 2025 May Submission136 Authors

Abstract: We propose \textbf{MT2ST}, a general and efficient framework for accelerating multi-task training by progressively transitioning to single-task optimization. Unlike conventional multi-task learning (MTL) or single-task fine-tuning (STL), MT2ST dynamically adjusts the training focus via two complementary strategies: \textit{Diminish}, which gradually down-weights auxiliary losses, and \textit{Switch}, which explicitly switches to the primary task at a scheduled point. We demonstrate the effectiveness of MT2ST across three key paradigms: representation learning, transformers, and diffusion models, covering both unimodal (text/image) and multimodal (vision-language) tasks. Extensive experiments show that MT2ST significantly improves training efficiency—achieving up to 56\% FLOPs compression—while maintaining or surpassing task performance. These results suggest MT2ST as a general-purpose solution for scalable and adaptive multi-task training.

Paper Type: Long

Research Area: Information Extraction

Research Area Keywords: Multi-Task Learning; Efficient ML

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 136