Improved generalization by noise enhancement

Takashi Mori, Masahito Ueda

Keywords: deep learning, generalization, stochastic gradient descent, large-batch training

Abstract: Recent studies have demonstrated that noise in stochastic gradient descent (SGD) is closely related to generalization; A larger SGD noise, if not too large, results in better generalization. Since the covariance of the SGD noise is proportional to $\eta^2/B$, where $\eta$ is the learning rate and $B$ is the minibatch size of SGD, so far the SGD noise has been controlled by changing $\eta$ and/or $B$. However, too large $\eta$ results in instability in the training dynamics and a small $B$ prevents scalable parallel computation. It is thus desirable to develop a method of controlling the SGD noise without changing $\eta$ and $B$. In this paper, we propose a method that achieves this goal using ``noise enhancement'', which is easily implemented in practice. We expound the underlying theoretical idea and demonstrate that the noise enhancement actually improves generalization for real datasets. It turns out that large-batch training with the noise enhancement even shows better generalization compared with small-batch training.

One-sentence Summary: We propose a simple yet efficient technique called "noise enhancement", which improves generalization by enhancing noise of the stochastic gradient descent.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=p-YjtgPPQi