Selecting Pre-trained Models for Transfer Learning with Data-centric Meta-features

Matt van den Nieuwenhuijzen, Carola Doerr, Henry Gouk, Jan N. van Rijn

Keywords: transfer learning, meta learning, meta features

TL;DR: Selecting pre-trained models for transfer learning with data-centric meta-features

Abstract: When applying a neural network to address a new learning problem, it is common to not train the network from scratch, but instead start with a neural network that has already been trained on a related dataset, and then fine-tune this on the data of the target task. This poses the question: which pre-trained network should be selected? In this work, we investigate this problem in the context of three different dataset relationships: same-source, same-domain, and cross-domain. We utilize Meta-Album, which offers an extensive collection of datasets from various unrelated domains. We first split each of the 30 datasets of Meta-Album into a meta-train dataset and meta-test dataset, then create pre-trained models for each meta-train dataset, and finally compare the performances of the pre-trained models in a fine-tuning context when applied to meta-test tasks. We categorize the performances into the three dataset relationship groups and find that the same-source category has the best performance. Then, using meta-features of the meta-train dataset and meta-test tasks, we train statistical meta-models that are employed to select the best pre-trained model for a given meta-test task. Our best meta-model identifies the best-performing model in $\sim 25$% of cases. It improves upon a baseline that always selects the best average model by more than $30$%.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.

Evaluation Metrics: No

Submission Number: 3