XAutoLLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML

ACL ARR 2025 February Submission1989 Authors

Abstract:

Experts in machine learning distinguish themselves from amateurs by leveraging domain knowledge to effectively navigate the myriad decisions involved in model selection, hyperparameter optimisation, and resource allocation. This distinction is especially critical for Language Models (LMs), whose repeated fine-tuning trials incur substantial computational overhead and raise environmental concerns. Yet, no single AutoML framework simultaneously addresses both model selection and hyperparameter optimisation for resource-efficient LM fine-tuning.

We propose XAutoLLM, a novel AutoML framework that integrates meta-learning to warm start the search space. By drawing on task- and system-level meta-features, XAutoLLM reuses insights from previously tuned LMs on related tasks. Through extensive experimentation on four text classification datasets, our framework discovers solutions in up to half the search time, reduces search errors by as much as sevenfold, and produces over 40% more pipelines that achieve improved performance–time trade-offs than a robust baseline. By systematically retaining and learning from prior experiments, XAutoLLM enables resource-friendly, Green AI fine-tuning, thereby fostering sustainable NLP pipelines that balance state-of-the-art outcomes with minimised computational overhead.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: Machine Learning for NLP, Language Modeling, Efficient/Low-Resource Methods for NLP

Contribution Types: Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 1989