Dataset Efficient Training with Model Ensembling

Yeonju Ro, Cong Xu, Agnieszka Ciborowska, Suparna Bhattacharya, Frankie Li, Martin Foltin

Abstract: We propose a dataset-efficient deep learning training method by ensembling multiple models trained on different subsets. The ensembling method leverages the difficulty level of data samples to select subsets that are representative and diverse. The approach involves building a common base model with a random subset of data and then allotting different subsets to the models in an ensemble. The models are trained with their own subsets and then merged into a single model. We design an multi-phase training strategy that aggregates models in the ensemble more frequently and prevents divergence. The experiments on ResNet18 and ImageNet show that ensembling outperforms the no-ensemble case and achieves 64.8% accuracy with only 30% of dataset, saving 20 hours of training time in a single V100 GPU training experiment with a mild accuracy drop.