Three Mechanisms of Weight Decay Regularization

Guodong Zhang, Chaoqi Wang, Bowen Xu, Roger Grosse

Abstract:

Weight decay is one of the standard tricks in the neural network toolbox, but the
reasons for its regularization effect are poorly understood, and recent results have
cast doubt on the traditional interpretation in terms of L2 regularization. Literal
weight decay has been shown to outperform L2 regularization for optimizers for
which they differ. We empirically investigate weight decay for three optimization
algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We
identify three distinct mechanisms by which weight decay exerts a regularization
effect, depending on the particular optimization algorithm and architecture: (1)
increasing the effective learning rate, (2) approximately regularizing the inputoutput Jacobian norm, and (3) reducing the effective damping coefficient for
second-order optimization. Our results provide insight into how to improve the
regularization of neural networks.