Deep GNN-driven Surrogate for the Better Meta-learning in AutoML

Nikolay Nikitin, Konstantin Cherniak, Mardyshkin Rostislav Romanovich, Egor Shikov, Peter Shevchenko, Illarion L. Iov, Maiia Pinchuk, Grigorii Kirgizov, Andrey Stebenkov, Anna Kalyuzhnaya

Keywords: GNN; Automl; meta-learning

TL;DR: Hybridization of GNN and EA can improve efficiency of Meta-AutoML for pipelines with variable-shaped heterogeneous structure

Abstract:

Automated Machine Learning (AutoML) aims to accelerate the process of solving machine learnging (ML) problems by providing tools for automating pipeline design. However, existing AutoML approaches are often computationally expensive, as they solve high-dimensional optimization tasks without leveraging knowledge from past solutions. Meta-learning aims to leverage past experience to improve the efficiency of solving new machine learning problems. In the context of automated pipeline design, meta-learning can facilitate the ranking of candidate pipelines by drawing structural insights from a database of previously solved tasks. However, existing meta-learning approaches tend to focus on relatively simple pipelines and tasks.

In this paper, we propose using Graph Neural Networks (GNNs) as probabilistic ranking surrogates for evolutionary optimization of pipelines with variable structures in AutoML. The GNNs are trained on meta-knowledge from a database of tabular classification problems to efficiently rank candidate pipelines based on their expected performance. This enables stronger initial estimates for optimization and accelerates convergence by leveraging surrogate evaluation of the fitness function. Our approach is implemented as an open-source library that can enhance the performance of state-of-the-art AutoML solutions.

Submission Number: 14