Data augmentation instead of explicit regularization

Alex Hernández-García, Peter König

Abstract: Modern deep artificial neural networks have achieved impressive results through models with very large capacity---compared to the number of training examples---that control overfitting with the help of different forms of regularization. Regularization can be implicit, as is the case of stochastic gradient descent or parameter sharing in convolutional layers, or explicit. Most common explicit regularization techniques, such as dropout and weight decay, reduce the effective capacity of the model and typically require the use of deeper and wider architectures to compensate for the reduced capacity. Although these techniques have been proven successful in terms of results, they seem to waste capacity. In contrast, data augmentation techniques reduce the generalization error by increasing the number of training examples and without reducing the effective capacity. In this paper we systematically analyze the effect of data augmentation on some popular architectures and conclude that data augmentation alone---without any other explicit regularization techniques---can achieve the same performance or higher as regularized models, especially when training with fewer examples.

TL;DR: In a deep convolutional neural network trained with sufficient level of data augmentation, optimized by SGD, explicit regularizers (weight decay and dropout) might not provide any additional generalization improvement.

Keywords: deep learning, data augmentation, regularization

Code: [ alexhernandezgarcia/data-aug-invariance](https://github.com/alexhernandezgarcia/data-aug-invariance) + [ 1 community implementation](https://paperswithcode.com/paper/?openreview=ByJWeR1AW)

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [CIFAR-100](https://paperswithcode.com/dataset/cifar-100)

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/data-augmentation-instead-of-explicit/code)