Data augmentation as stochastic optimization

Boris Hanin, Yi Sun

Keywords: data augmentation, stochastic optimization, scheduling, convex optimization, overparametrization

Abstract: We present a theoretical framework recasting data augmentation as stochastic optimization for a sequence of time-varying proxy losses. This provides a unified language for understanding techniques commonly thought of as data augmentation, including synthetic noise and label-preserving transformations, as well as more traditional ideas in stochastic optimization such as learning rate and batch size scheduling. We then specialize our framework to study arbitrary augmentations in the context of a simple model (overparameterized linear regression). We extend in this setting the classical Monro-Robbins theorem to include augmentation and obtain rates of convergence, giving conditions on the learning rate and augmentation schedule under which augmented gradient descent converges. Special cases give provably good schedules for augmentation with additive noise, minibatch SGD, and minibatch SGD with noise.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We develop a principled theoretical approach relating data augmentation to stochastic optimization and apply it to obtain provably good augmentation schedules in the overparametrized linear setting.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=Rn4DrA3bD0