Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, Ameet Talwalkar

Keywords: optimization, trajectory, stability, sharpness, implicit bias, implicit regularization, L-smoothness, deep learning theory, science of deep learning

Abstract: We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the value $2 / \text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability.

One-sentence Summary: We trained neural networks using full-batch gradient descent -- you won't believe what happens next!

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ locuslab/edge-of-stability](https://github.com/locuslab/edge-of-stability)

Data: [WikiText-2](https://paperswithcode.com/dataset/wikitext-2)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2103.00065/code)