Learning-to-Rank for AutoML: A Simple and Robust Alternative to Score-Based Pipeline Selection

Hernan Ceferino Vazquez, Jorge Sánchez, Verónica Bogado, Pucci Romero Tobias

Confirmation: our paper adheres to reproducibility best practices. In particular, we confirm that all important details required to reproduce results are described in the paper,, the authors agree to the paper being made available online through OpenReview under a CC-BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), and, the authors have read and commit to adhering to the AutoML 2025 Code of Conduct (https://2025.automl.cc/code-of-conduct/).

TL;DR: This paper proposes a ranking-based framework for selecting machine learning pipelines in AutoML, offering a simpler, more robust alternative to traditional performance-prediction methods by focusing on relative rankings instead of absolute scores.

Abstract: This paper introduces a learning-to-rank (LTR) framework to address the problem of pipeline selection in automated machine learning (AutoML) systems. Traditional approaches typically involve learning to predict the performance of various pipelines on a given task based on data acquired from previous tasks, i.e. meta-learning, which can be complex due to the need for task-specific regression models and performance metrics. Instead, the proposed framework focuses on predicting the relative ranking of pipelines, aligning more directly with the ultimate goal of pipeline selection. An important advantage of this approach is that it only requires a reformulation of the input space—replacing absolute performance scores with rank positions—without modifying the underlying model architecture. This makes the method highly general and compatible with a wide range of existing AutoML techniques. In addition, through controlled experiments, we show that ranking-based models are significantly less sensitive to noisy or overfitted meta-learning data, a common issue in practical AutoML settings. As a result, it enables more robust, metric-agnostic solutions and facilitates evaluation through ranking metrics such as NDCG and MRR. The framework is evaluated on public OpenML datasets, showing consistent advantages for ranking-based models. Furthermore, we explore its integration with Bayesian optimization and Monte Carlo Tree Search, yielding improved results in ranking quality. Finally, we find a strong relationship between ranking-based metrics and key AutoML objectives such as final performance score and time-to-solution, highlighting the potential of rank-based approaches to enhance AutoML systems.

Submission Number: 30