Unique Properties of Flat Minima in Deep Networks

Rotem Mulayoff, Tomer Michaeli

Abstract: It is well known that (stochastic) gradient descent has an implicit bias towards flat minima. In deep neural network training, this mechanism serves to screen out minima. However, the precise effec...