## AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights

PyTorch implementation of AdamP and SGDP optimizers

Anonymous Author(s)

### Abstract

Normalization techniques, such as batch normalization (BN), are a boon for modern deep learning. They let weights converge more quickly with often better generalization performances. It has been argued that the normalization-induced scale invariance among the weights provides an advantageous ground for gradient descent (GD) optimizers: the effective step sizes are automatically reduced over time, stabilizing the overall training procedure. It is often overlooked, however, that the additional introduction of momentum in GD optimizers results in a far more rapid reduction in effective step sizes for scale-invariant weights, a phenomenon that has not yet been studied and may have caused unwanted side effects in the current practice. This is a crucial issue because arguably the vast majority of modern deep neural networks consist of (1) momentum-based GD (e.g. SGD or Adam) and (2) scale-invariant parameters (e.g. more than 90% of the weights in ResNet are scale-invariant due to BN). In this paper, we verify that the widely-adopted combination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances. We propose a simple and effective remedy, SGDP and AdamP: get rid of the radial component, or the norm-increasing direction, at each optimizer step. Because of the scale invariance, this modification only alters the effective step sizes without changing the effective update directions, thus enjoying the original convergence properties of GD optimizers. Given the ubiquity of momentum GD and scale invariance in machine learning, we have evaluated our methods against the baselines on 13 benchmarks. They range from vision tasks like classification (e.g. ImageNet), retrieval (e.g. CUB and SOP), and detection (e.g. COCO) to language modelling (e.g. WikiText) and audio classification (e.g. DCASE) tasks. We verify that our solution brings about uniform gains in performances in those benchmarks. Source code will be available at public-url.

## How it works?

Please see the attached html `project_page\index.html` [link](../project_page/index.html).


## Getting Started

Usage is exactly same as [torch.optim](https://pytorch.org/docs/stable/optim.html) library!

```python
from adamp import AdamP
optimizer = AdamP(lr=0.001, betas=(0.9, 0.999), weight_decay=1e-2)
```

```python
from sgdp import SGDP
optimizer = SGDP(lr=0.1, weight_decay=1e-5, momentum=0.9, nesterov=True)
```

## Arguments
`SGDP` and `AdamP` share arguments with [torch.optim.SGD](https://pytorch.org/docs/stable/optim.html#torch.optim.SGD) and [torch.optim.Adam](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam).
There are two additional hyperparameters; we recommend using the default values.
- `delta` : threhold that determines whether a set of parameters is scale invariant or not (default: 0.1)
- `wd_raito` : relative weight decay applied on _scale-invariant_ parameters compared to that applied on _scale-variant_ parameters (default: 0.1)