Data-Efficient Augmentation for Training Neural Networks

Tian Yu Liu, Baharan Mirzasoleiman

Keywords: Data Augmentation, Neural Network, Coresets

Abstract: Data augmentation is essential to achieve state-of-the-art performance in many deep learning applications. However, modern data augmentation techniques become computationally prohibitive for large datasets. To address this, we propose a rigorous technique to select subsets of data points that when augmented, closely capture the training dynamics of full data augmentation. We first show that data augmentation, modeled as additive perturbations, speeds up learning by enlarging the smaller singular values of the network Jacobian. Then, we propose a framework to iteratively extract small subsets of training data that when augmented, closely capture the alignment of the fully augmented Jacobian with label/residual vector. We prove that stochastic gradient descent applied to augmented subsets found by our approach have similar training dynamics to that of fully augmented data. Our experiments demonstrate that our method outperforms state-of-the-art max-loss strategy by 7.7% on CIFAR10 while achieving 6.3x speedup, and by 4.7% on SVHN while achieving 2.2x speedup, using 10% and 30% subsets, respectively.

One-sentence Summary: We theoretically analyze data augmentation and propose a rigorous technique to select subsets of data points that when augmented provide a similar speedup and generalization performance as that of full data augmentation.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2210.08363/code)