Implicit bias of gradient descent on convolutional generatorsDownload PDF

17 May 2019 (modified: 05 May 2023)Submitted to ICML Deep Phenomena 2019Readers: Everyone
Keywords: Implicit bias, inductive bias, convolutional neural networks
Abstract: The majority of image generating neural networks have a convolutional architecture, and this architecture has empirically been proven to be efficient at representing natural images. It is widely acknowledged that convolutional neural networks incorporate strong prior assumptions about natural images, sometimes referred to as an inductive or implicit bias. An instantiation of this bias towards simple, natural images, is that an un-trained overparmeterized convolutional network optimized with gradient descent fits a single natural image significantly faster than complex images such as noise. In this paper, we isolate this effect and demonstrate that it is most pronounced if the network incorporates convolutions with a fixed and smooth upsampling kernel. This effect can be understood as the network architecture imposing constraints so that an natural image is closer in parameter space to a random initialization than a highly complex image such as noise.
1 Reply

Loading