In-Context Meta-Learning with Large Language Models for Automated Model and Hyperparameter Selection

Youssef Attia El Hili, Albert Thomas, Abdelhakim Benechehab, Corentin Léger, Corinne Ancourt, Balázs Kégl

Keywords: Large Language Models, In-Context Learning, Hyperparameter Optimization, Meta-Learning

TL;DR: This paper explores using large language models as in-context meta-learners for model and hyperparameter selection in tabular classification and regression tasks.

Abstract: Model and hyperparameter selection is a critical yet costly step in machine learning, often requiring expert intuition or extensive search. We investigate whether large language models (LLMs) can reduce this cost by acting as in-context meta-learners that generalize across tasks to propose effective model-hyperparameter choices without iterative optimization. Each task is represented as structured metadata, and we prompt an LLM under two strategies: Zero-Shot, using only the target task metadata, and Meta-Informed, which augments the prompt with metadata–recommendation pairs from prior tasks. Evaluated on 22 tabular Kaggle challenges, Meta-Informed prompting outperforms Zero-Shot and hyperparameter optimization baselines, approaching expert AutoML blends while yielding interpretable reasoning traces and efficiency gains under tight training budgets. These results suggest that LLMs can transfer knowledge across tasks to guide automated model selection, establishing model and hyperparameter selection as a concrete testbed for studying emergent adaptation beyond language domains.

Submission Number: 71