Selecting Pre-trained Models for Transfer Learning with Data-centric Meta-features

Published: 12 Jul 2024, Last Modified: 09 Aug 2024AutoML 2024 WorkshopEveryoneRevisionsBibTeXCC BY 4.0
Keywords: transfer learning, meta learning, meta features
TL;DR: Selecting pre-trained models for transfer learning with data-centric meta-features
Abstract: When applying a neural network to address a new learning problem, it is common to not train the network from scratch, but instead start with a neural network that has already been trained on a related dataset, and then fine-tune this on the data of the target task. This poses the question: which pre-trained network should be selected? In this work, we investigate this problem in the context of three different dataset relationships: same-source, same-domain, and cross-domain. We utilize Meta-Album, which offers an extensive collection of datasets from various unrelated domains. We first split each of the 30 datasets of Meta-Album into a meta-train dataset and meta-test dataset, then create pre-trained models for each meta-train dataset, and finally compare the performances of the pre-trained models in a fine-tuning context when applied to meta-test tasks. We categorize the performances into the three dataset relationship groups and find that the same-source category has the best performance. Then, using meta-features of the meta-train dataset and meta-test tasks, we train statistical meta-models that are employed to select the best pre-trained model for a given meta-test task. Our best meta-model identifies the best-performing model in $\sim 25$% of cases. It improves upon a baseline that always selects the best average model by more than $30$%.
Submission Checklist: Yes
Broader Impact Statement: Yes
Paper Availability And License: Yes
Code Of Conduct: Yes
Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.
Evaluation Metrics: No
Submission Number: 3
Loading