Simplifying Adam: Bias Correction Debunked

Sam Laing, Antonio Orvieto

Keywords: Transformer Pretraining, Adam Optimizer, Deep Learning

TL;DR: We investigate whether the bias correction term can be safely removed from Adam when pretraining GPT2-style language models.

Abstract: The Adam optimizer is a cornerstone of modern deep learning, yet the empirical necessity of each of its individual components is often taken for granted. This paper presents a focused investigation into the role of bias-correction, a feature whose contribution remains poorly understood. Through a series of systematic ablations on vision and language modelling tasks, we demonstrate that the conventional wisdom surrounding bias correction is misleading. In particular, we demonstrate that in the optimal hyper-parameter configuration, the inclusion of bias correction leads to no improvement in final test performance. Moreover, unless appropriate learning rate scheduling is implemented, the inclusion of bias correction can sometimes be detrimental to performance. We further reinterpret bias correction as a form of implicit learning rate scheduling whose behaviour is strongly dependent on the choice of smoothing hyper-parameters $\beta_1, \beta_2 \in [0,1)$. Our findings challenge the universal inclusion of this component.

Submission Number: 35