How Important is Importance Sampling for Deep Budgeted Training?

Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, Kevin McGuinness

Keywords: Budgeted training, importance sampling, data augmentation, deep learning

Abstract: Long iterative training processes for Deep Neural Networks (DNNs) are commonly required to achieve state-of-the-art performance in many computer vision tasks. Core-set selection and importance sampling approaches might play a key role in budgeted training regimes, i.e.~when limiting the number of training iterations. The former demonstrate that retaining informative samples is important to avoid large drops in accuracy, and the later aim at dynamically estimating the sample importance to speed-up convergence. This work explores this paradigm and how a budget constraint interacts with importance sampling approaches and data augmentation techniques. We show that under budget restrictions, importance sampling approaches do not provide a consistent improvement over uniform sampling. We suggest that, given a specific budget, the best course of action is to disregard the importance and introduce adequate data augmentation. For example, training in CIFAR-10/100 with 30% of the full training budget, a uniform sampling strategy with certain data augmentation surpasses the performance of 100% budget models trained with standard data augmentation. We conclude from our work that DNNs under budget restrictions benefit greatly from variety in the samples and that finding the right samples to train is not the most effective strategy when balancing high performance with low computational requirements. The code will be released after the review process.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Explore the interactions between importance sampling and data augmentation for budgeted training

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2110.14283/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=IBda4ryy4L