Gradient Origin Networks

Sam Bond-Taylor, Chris G. Willcocks

Keywords: Deep Learning, Generative Models, Implicit Representation

Abstract: This paper proposes a new type of generative model that is able to quickly learn a latent representation without an encoder. This is achieved using empirical Bayes to calculate the expectation of the posterior, which is implemented by initialising a latent vector with zeros, then using the gradient of the log-likelihood of the data with respect to this zero vector as new latent points. The approach has similar characteristics to autoencoders, but with a simpler architecture, and is demonstrated in a variational autoencoder equivalent that permits sampling. This also allows implicit representation networks to learn a space of implicit functions without requiring a hypernetwork, retaining their representation advantages across datasets. The experiments show that the proposed method converges faster, with significantly lower reconstruction error than autoencoders, while requiring half the parameters.

One-sentence Summary: A new model that uses the negative gradient of the loss with respect to the origin as a latent vector is found to be superior to equivalent networks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ cwkx/GON](https://github.com/cwkx/GON) + [ 3 community implementations](https://paperswithcode.com/paper/?openreview=0O_cQfw6uEh)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2007.02798/code)