Augment your batch: better training with larger batches

Elad Hoffer, Itay Hubara, Niv Giladi, Daniel Soudry

Abstract:

Recently, there is regained interest in large batch training of neural networks, both of theory and practice. New insights and methods allowed certain models to be trained using large batches with no adverse impact on performance. Most works focused on accelerating wall clock training time by modifying the learning rate schedule, without introducing accuracy degradation.
We propose to use large batch training to boost accuracy and accelerate convergence by combining it with data augmentation. Our method, "batch augmentation", suggests using multiple instances of each sample at the same large batch. We show empirically that this simple yet effective method improves convergence and final generalization accuracy. We further suggest possible reasons for its success.

Keywords: Large Batch Training, Augmentation, Deep Learning

TL;DR: Improve accuracy by large batches composed of multiple instances of each sample at the same batch

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1901.09335/code)