Improving Transfer Learning by means of Ensemble Learning and Swarm Intelligence-based Neuroevolution

Adri Gómez, Monica Abella, Manuel Desco

Keywords: Ensemble Learning, Deep Learning, Neural Architecture Search, Swarm Intelligence

TL;DR: Assess the interaction of Ensemble Deep Learning and Neural Architecture Search when the models generated overfit to the model selection partition.

Abstract: Neural Architecture Search (NAS) methods, when applied to very small but complex datasets, tend to overfit on the validation partitions and underperform compared to Transfer Learning models. In order to reduce the bias and variance of their predictions, Deep Ensemble Learning (DEL) can be used. The combination of NAS and DEL has only been employed on large datasets in the literature, but these scenarios do not present the overfitting in validation we typically experience, for instance, on medical imaging applications. In this work, we empirically assess the feasibility of NAS, DEL and the combination of the two on both small and large dataset scenarios. We find that the performance of the ensembles highly depend on the degree of overfitting of the standalone models, but always will outperform the worst generalizing models in the population.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Optional Meta-Data For Green-AutoML: This blue field is just for structuring purposes and cannot be filled.

Evaluation Metrics: Yes

Submission Number: 9