Implicit bias of gradient descent on convolutional generators

May 17, 2019 Blind Submission readers: everyone
  • Keywords: Implicit bias, inductive bias, convolutional neural networks
  • Abstract: The majority of image generating neural networks have a convolutional architecture, and this architecture has empirically been proven to be efficient at representing natural images. It is widely acknowledged that convolutional neural networks incorporate strong prior assumptions about natural images, sometimes referred to as an inductive or implicit bias. An instantiation of this bias towards simple, natural images, is that an un-trained overparmeterized convolutional network optimized with gradient descent fits a single natural image significantly faster than complex images such as noise. In this paper, we isolate this effect and demonstrate that it is most pronounced if the network incorporates convolutions with a fixed and smooth upsampling kernel. This effect can be understood as the network architecture imposing constraints so that an natural image is closer in parameter space to a random initialization than a highly complex image such as noise.
0 Replies