Theoretical Analysis of Consistency Regularization with Limited Augmented Data

Shuo Yang, Yijun Dong, Rachel Ward, Inderjit S Dhillon, sujay sanghavi, Qi Lei

Keywords: data augmentation, consistency regularization, generalization bound

Abstract: Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data. In this paper, we take a small step in this direction; we present a simple new statistical framework to analyze data augmentation - specifically, one that captures what it means for one input sample to be an augmentation of another, and also the richness of the augmented set. We use this to interpret consistency regularization as a way to reduce function class complexity, and characterize its generalization performance. Specializing this analysis for linear regression shows that consistency regularization has strictly better sample efficiency as compared to empirical risk minimization on the augmented set. In addition, we also provide generalization bounds under consistency regularization for logistic regression and two-layer neural networks. We perform experiments that make a clean and apples-to-apples comparison (i.e. with no extra modeling or data tweaks) between ERM and consistency regularization using CIFAR-100 and WideResNet; these demonstrate the superior efficacy of consistency regularization.

One-sentence Summary: We develop a theoretical framework to understand the statistical efficiency of consistency regularization with limited data augmentations.