δ-SAM: Sharpness-Aware Minimization with Dynamic Reweighting

Anonymous

Abstract: Deep neural networks are often overparameterized and may not easily achieve model generalization. Adversarial training has shown effectiveness in improving generalization by regularizing the change of loss on top of adversarially chosen perturbations. The recently proposed sharpness-aware minimization (SAM) algorithm conducts adversarial weight perturbation, encouraging the model to converge to a flat minima. Unfortunately, due to increased computational cost, adversarial weight perturbation can only be efficiently estimated per-batch instead of per-instance by SAM, leading to degraded performance. In this paper, we tackle this efficiency bottleneck and propose the first instance-based weight perturbation method: sharpness-aware minimization with dynamic reweighting (δ-SAM). δ-SAM dynamically reweights perturbation within each batch by estimated guardedness (i.e. unguarded instances are up-weighted), serving as a better approximation to per-instance perturbation. Experiments on various tasks demonstrate the effectiveness of δ-SAM.

Paper Type: short