How Learning Dynamics Drive Adversarially Robust Generalization?

ICLR 2026 Conference Submission2047 Authors

Keywords: Adversarially Robust Generalization, Robust Overfitting, PAC-Bayesian Framework, SGD Dynamics

Abstract: Despite significant progress in adversarially robust learning, the underlying mechanisms that govern robust generalization remain poorly understood. We propose a novel PAC-Bayesian framework that explicitly links adversarial robustness to the posterior covariance of model parameters and the curvature of the adversarial loss landscape. By characterizing discrete-time SGD dynamics near a local optimum under quadratic loss, we derive closed-form posterior covariances for both the stationary regime and the early phase of non-stationary transition. Our analyses reveal how key factors, such as learning rate, gradient noise, and Hessian structure, jointly shape robust generalization during training. Through empirical visualizations of these theoretical quantities, we fundamentally explain the phenomenon of robust overfitting and shed light on why flatness-promoting techniques like adversarial weight perturbation help to improve robustness.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 2047