SALR: Sharpness-aware Learning Rates for Improved Generalization

Xubo Yue, Maher Nouiehed, Raed Al Kontar

Keywords: Loss-surface, sharpness, learning rate, generalization

Abstract: In an effort to improve generalization in deep learning, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: A Sharpness-aware Learning Rate Framework.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=jYhFkPwaHq