Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems

Bingcong Li, Liang Zhang, Niao He

Keywords: sharpness-aware minimization, LoRA, implicit regularization, finetuning, language models

TL;DR: We leverage implicit regularization to enhance the computational efficiency of sharpness-aware minimization (SAM).

Abstract:

Sharpness-aware minimization (SAM) improves generalization of various deep learning tasks. Motivated by popular architectures such as LoRA, we explore the implicit regularization of SAM for scale-invariant problems involving two groups of variables. Instead of focusing on commonly used sharpness, this work introduces a concept termed *balancedness*, defined as the difference between the squared norm of two variables. This allows us to depict richer global behaviors of SAM. In particular, our theoretical and empirical findings reveal that i) SAM promotes balancedness; and ii) the regularization on balancedness is *data-responsive* -- outliers have stronger impact.
The latter coincides with empirical observations that SAM outperforms SGD in the presence of outliers.
Leveraging the implicit regularization, we develop a resource-efficient SAM variant, balancedness-aware regularization (BAR), tailored for scale-invariant problems such as finetuning language models with LoRA. BAR saves 95% computational overhead of SAM, with enhanced test performance across various tasks on RoBERTa, GPT2, and OPT-1.3B.

Primary Area: Optimization for deep networks

Submission Number: 6575