On Linear Stability of SGD and Input-Smoothness of Neural NetworksDownload PDF

21 May 2021, 20:45 (edited 21 Jan 2022)NeurIPS 2021 SpotlightReaders: Everyone
  • Keywords: Stochastic Gradient Descent, Implicit regularization, Linear stability, Sobolev Seminorm
  • TL;DR: On Linear Stability of SGD and Input-Smoothness of Neural Networks
  • Abstract: The multiplicative structure of parameters and input data in the first layer of neural networks is explored to build connection between the landscape of the loss function with respect to parameters and the landscape of the model function with respect to input data. By this connection, it is shown that flat minima regularize the gradient of the model function, which explains the good generalization performance of flat minima. Then, we go beyond the flatness and consider high-order moments of the gradient noise, and show that Stochastic Gradient Dascent (SGD) tends to impose constraints on these moments by a linear stability analysis of SGD around global minima. Together with the multiplicative structure, we identify the Sobolev regularization effect of SGD, i.e. SGD regularizes the Sobolev seminorms of the model function with respect to the input data. Finally, bounds for generalization error and adversarial robustness are provided for solutions found by SGD under assumptions of the data distribution.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/ChaoMa93/Sobolev-Reg-of-SGD
10 Replies

Loading