Decoupling Generalization and Adaptation in Meta-Learning for Large Language Models

ACL ARR 2026 January Submission5339 Authors

Keywords: meta-learning, large language models, generalization, task adaptation, reinforcement learning

Abstract: Fine-tuning large language models (LLMs) for downstream tasks remains expensive, even with parameter-efficient methods like Low-Rank Adaptation (LoRA). In this regard, meta-learning approaches such as Model-Agnostic Meta-Learning for LLMs (MAML-en-LLM) and Amortized Bayesian Meta-Learning for LoRA (ABMLL) have emerged as promising solutions for rapid downstream LLM adaptation. However, these methods fundamentally couple two distinct objectives: learning generalizable initializations and enabling efficient task adaptation. We argue that this coupling limits both the quality of learned representations and adaptation efficiency. In this paper, we introduce DeGAML-LLM (Decoupled Generalization and Adaptation Meta-Learning for LLMs), a novel framework that explicitly separates these two objectives through dedicated parameter spaces. Specifically, we maintain a generalization module that learns task-agnostic representations across the task distribution, and an adaptation module that specializes in rapid task-specific adjustment. Extensive experiments on common-sense reasoning, mathematics, logic, social, medical and coding benchmarks across model scales demonstrate that DeGAML-LLM outperforms existing meta-learning and standard multi-task baselines.

Paper Type: Short

Research Area: Machine Learning for NLP

Research Area Keywords: meta learning; multi-task learning; transfer learning / domain adaptation; representation learning; generalization; reinforcement learning

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 5339