Sharpness-aware Minimization for Efficiently Improving GeneralizationDownload PDF

Published: 12 Jan 2021, Last Modified: 28 May 2023ICLR 2021 SpotlightReaders: Everyone
Keywords: Sharpness Minimization, Generalization, Regularization, Training Method, Deep Learning
Abstract: In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by the connection between geometry of the loss landscape and generalization---including a generalization bound that we prove here---we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-{10, 100}, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels.
One-sentence Summary: Motivated by the connection between geometry of the loss landscape and generalization, we introduce a procedure for simultaneously minimizing loss value and loss sharpness.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Code: [![github](/images/github_icon.svg) google-research/sam]( + [![Papers with Code](/images/pwc_icon.svg) 10 community implementations](
Data: [Birdsnap](, [CIFAR-10](, [CIFAR-100](, [FGVC-Aircraft](, [Fashion-MNIST](, [Food-101](, [ImageNet](, [Oxford 102 Flower](, [SVHN](, [Stanford Cars](
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 2 code implementations](
24 Replies